DOCUMENT RESUME 



ED 453 066 


SE 064 719 


AUTHOR 


Wiltz, L. Kate 


TITLE 


Proceedings of the Teton Summit for Program Evaluation in 
Nonformal Environmental Education (Kelly, Wyoming, May 
19-22, 2000) . 


INSTITUTION 


Teton Science School, Kelly, WY. ; Ohio State Univ., 
Columbus . - - 


SPONS AGENCY 


Fish and Wildlife Service (Dept, of Interior), Washington, 
DC.; National Park Foundation, Washington, DC. 


PUB DATE 


2001-00-00 


NOTE 


4 7 p . ; Also supported by the Brained Foundation. Photographs 
by Claudia Figueiredo and L. Kate Wiltz. Resource list 
complied by Claudia Figueiredo. Graphic design by John K. 
Victor . 


PUB TYPE 


Collected Works - Proceedings (021) 


EDRS PRICE 


MF01/PC02 Plus Postage. 


DESCRIPTORS 


Cost Effectiveness; Economics; * Environmental Education; 
Interdisciplinary Approach; *Nonformal Education; Problem 
Solving; ^Program Evaluation 


ABSTRACT 


This document contains the proceedings of the Teton Summit 



Conference for Program Evaluation in Nonformal Environmental Education. The 
Summit was co-hosted by the Teton Science School (TSS) and The Ohio State 
University with the goal of improving efficacy of evaluation in environmental 
education. Contents include: (1) What Is Evaluation? (2) For Whom Is 
Evaluation Done? (3) Why Evaluate? (4) How Is Evaluation Done? (5) What Is 
and What Should Be? (6) The Politics of Evaluation (7) What Environmental 
Education Evaluation Can Look Like (8) P.A.R.K.S. as a Model; (9) Valuing 
Programs from an Economic Perspective: Cost Analysis in Environmental 
Education Evaluation; (10) Cost-Benefit and Cost-Effectiveness; (11) Other 
Factors When Considering Costs; (12) Should Environmental Education Program 
Evaluation Be Required? (13) Possibilities: Where Do We Go From Here? (14) 
Matching; (15) Evaluation and Research; (16) A Culture of Evaluation; and 
(17) Identifying the "It" in Evaluation. Appendices include a list of summit 
participants, materials from the summit, and a resource guide. (YDS) 



o 

ERIC 



Reproductions supplied by EDRS are the best that can be made 
from the original document. 



VO 

VO 

o 



Proceedings 



o f 



CO 

in 

Tt 



Q 

w 



The 



Feton Summit 



for Program Evaluation in 








Text by L. Kate Wiltz 

Photographs by Claudia Figueiredo and L. Kate Waltz 
Resource List compiled by Claudia Figueiredo 
Graphic design by John K. Victor 



Table of Contents 



Preface 2 

Introduction 3 

I. Jack Shea 3 

II. Emmalou Norland and L. Kate Wiltz 3 

III. Michael Q. Patton 5 

Part I: The Journey to the Summit 9 

What is Evaluation? 9 

For Whom is Evaluation Done? 10 

Why Evaluate? 11 

How is Evaluation Done? 12 

What is and What Should Bel 5 

The Politics of Evaluation 16 

What Environmental Education Evaluation Can Look Like 17. 

P.A.R.K.S. as a Model 18 

Valuing Programs from an Economic Perspective: Cost Analysis in 

Environmental Education Evaluation 19 

Cost-Benefit and Cost-Effectiveness 20 

Other Factors when Considering Costs 21 

Should Environmental Education Program Evaluation be Required? ..22 

Possibilities: Where Do We Go From Here? 22 

Part II: Summit Take-Aways 25 

Matching 25 

Evaluation and Research 26 

A Culture of Evaluation 27 

Identifying the TL in Evaluation 28 

References in Text 30 

Appendices 30 

List of Summit Participants 30 

Materials from the Summit 31 

Resource Guide 32 



i 



Thanks to all who made 



the Summit and this publication 
possible . We could have accom- 
plished little without the support of 
those who saw the potential of this 
type of intellectual exploration: The 
U.S. Fish and Wildlife Service, The 
National Park Foundation, and The 
Brainerd Foundation. Also, many 
thanks to our facilitator, Michael 
Patton, whose insight inspired and 
led the conversations at the Summit 
and to the coordinators of the 
Summit, Emmalou Norland and 
Jack Shea. Finally, the careful 
documentation of the Summit 
discussions would have been 
impossible without the dedication to 
task and attention to detail of The 
Ohio State University graduate 
student team: Claudia Figueiredo, 
Liz Hall, Cindy Somers, and Gina 
Zwerling. 




T • H ■ E 



OHIO 

SPtfE 

UNIVERSITY 



i 



i 



O 

ERIC 



3 



i 




Preface 

The following text was written in order to 
capture the spirit of the conversations that 
took place over two and half days at the Teton 
Science School in Kelly, Wyoming, in May 
2000. I have chosen to reflect on the dialogues 
and presentations that took place at the 
Summit, and then to document their collective course. Over 
those few days, the participants in the Summit explored the 
topic of evaluation under the guidance of the Summit facilitator, 
Michael Patton. As he came to understand the concerns of his 
audience, and they came to some consensus on what evaluation 
means to them, the group of participants considered different 
ideas, issues and questions regarding their work in environmen- 
tal education. 




Although written by just one person, the words here belong 
to the entire group of Summit participants. By no means did we 
speak with one voice during the Summit - the conversations 
Teton Science School were more complex and considered many points that I am sure 

are not captured in these pages. However, 
several people have worked to capture the ideas 
and viewpoints expressed the weekend of May 
19-22, 2000. We have put together what we 
hope to be a coherent and cohesive discourse 
on the topic of environmental education 
evaluation, in order that this document may be 
of some use to others who wish to look into this 
emerging specialization. What follows is an 
interpretation of the journey we took last May - 
the Journey to the Summit. 
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Introduction I 

With the principle goal of improving the efficacy of evaluation in environ- 
mental education, the Teton Science School (TSS) and Ohio State University 
(OSU) co-hosted the Summit for Program Evaluators and Environmental 
Educators in May 2000, in Jackson Hole, WY. The concept of this summit 
originated at the June 1999 “Education for the Wild” symposium in 
Shepherdstown, W V. Sponsored by the Murie Center and the National Conserva- 
tion Training Center, U.S. Fish and Wildlife Service, this conference gathered 
professionals working in environmental education on public lands to create a 
forum of diverse expertise and experience. The TSS - OSU Summit was intended 
to be a “next-step” in the examination of evaluation in environmental education. 
The conference convened recognized leaders in the fields of non-formal environ- 
mental education and program evaluation in order to initiate a discussion on the 
best methods of determining strengths and weaknesses in environmental 
education. 

Information sharing, creative problem solving, and strategic planning 
allowed Summit participants to explore relevant issues surrounding environ- 
mental education evaluation, identify barriers to effective program evaluation, 
and begin the process of standardizing the criteria for evaluation. The ideas and 
strategies explored during the Summit are presented here in the event proceed- 
ings. Summit highlights included facilitated discussions by Michael Patton, a 
nationally recognized expert in evaluation research, use, and practice, and small 
learning groups discussing topics such as the relationship between evaluation 
and research, how to define evaluation and its purpose in educational program- 
ming, the costs and benefits involved with program evaluation alternatives, and 
the standards of practice in environmental education. 

Through the Summit, we hope to stimulate further examination of the 
evaluation issue. As responsible environmental educators and center adminis- 
trators, we are always trying to improve the efficacy of our programs. We are 
asked to justify and articulate their outcomes to parents, school systems, the 
community, and funders. Program evaluation and assessment are an essential 
point in the maturing process of schools and education centers. This Summit 
has created a platform on which relationships between environmental education 
and assessment experts can be built and around which future work can be 
centered. 

Jack Shea 



Introduction II 

We would like to take this space to provide some theoretical background to 
the Teton Summit and to highlight the need for the dialogue that was begun at 
the Teton Science School this spring. 

Environmental education is an interdisciplinary approach to learning how 
the natural world intersects with the social, political, and economic arenas 
within which humans exist. Environmental education and interpretation 
organizations around the world have proliferated in an effort to create an 
‘environmentally literate 7 populace. Until recently, however, many of these 
organizations have not had to ‘prove 7 that they were accomplishing their goals. 
But today, more and more funding organizations have begun to require evidence 
that targeted goals are being achieved. For example, the Government Perfor- 
mance and Results Act of 1993 has changed the focus in governmental account- 
ability from program monitoring and auditing to the employment of evaluation 
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processes to determine if the intended program outcomes are being achieved. 
This type of accountability is now also being required by non-governmental 
foundations. As a result, environmental education and interpretation agencies 
throughout the United States are under great pressure to provide evidence of 
positive outcomes related to the goals of their programming. This accountabil- 
ity in the form of program evaluation is then incorporated into the decision- 
making/planning process for future funding allocations. 

Program evaluation is the “systematic collection of information about the 
activities, characteristics, and outcomes of programs for use by specific people to 
reduce uncertainties, improve effectiveness, and make decisions with regard to 
what those programs are doing and affecting” (Patton, 1997). Over the past 30 
or more years, research in evaluation and program planning has created a large 
theory base on evaluative approaches and techniques. 

Concurrently, research in the field of environmental education has indi- 
cated that attention to a number of inputs in a learner's experience can lead to 
responsible environmental behavior (Hungerford and Volk, 1990), and many 
private and nonprofit agencies indicate such behavior as a desired outcome in 
their programming (Simmons, 1993). The linking of program inputs to the 
desired outcomes across visitors is key to both formative and summative 
evaluation. Equally important in evaluation, is the linking of appropriate 
outcomes to the program inputs. These outcomes are contextually, as well as 
educationally, based. 

In the formal environmental education arena, where virtually all studies in 
environmental education program evaluation have occurred, students are 
usually 'tested' for outcomes such as knowledge, skills, and behavior. Nonformal 
environmental education and interpretation that takes place on public and 
private land across the country differs from formal education endeavors in that 
these non-captive audiences are more heterogeneous and are viewed as clients or 
visitors rather than students. With nonformal environmental education in 
which program goals and objectives are frequently not as well defined and formal 
environmental education that occurs in nonformal settings, outcome measure- 
ment can be more challenging. 

The above factors indicate the need for increased interaction between 
environmental educators, funders with an interest in environmental program- 
ming, and professional program evaluators. There is an emerging interest in 
both fields for expanded interaction and exchange. The American Evaluation 
Association has established a topical interest group for its members interested in 
environmental program evaluation, and the North American Association for 
Environmental Education held a workshop on program evaluation at its 1999 
national conference. 

The primary purpose of the Teton Summit was to enhance program 
evaluation in environmental education by facilitating communication among 
professionals in nonformal environmental education, evaluation, and environ- 
mental education grantmaking. By sharing the experience and expertise of 
participants from the above fields, the summit worked toward elevating the 
quality and utility of evaluation in nonformal environmental education while 
maintaining the integrity of the evaluation profession. Michael Patton discussed 
the theory base and the standard approaches utilized in program evaluation. 
Researchers and evaluators working in environmental education added their 
shared their perspectives and strategies in evaluating environmental education 
programs across the country. Environmental educators delineated the unique 
needs and context of nonformal environmental education programming. Finally, 
researchers and representatives from granting foundations shared their broad- 
based experience and perspectives to enable effective communication and 
application of the ideas discussed. 
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There were four specific objectives of the three-day summit: 

1. To convene identified and interested leaders in the fields of nonformal 
environmental education and program evaluation and to initiate discussion on 
best practices for environmental education evaluation. 

2. To identify and explore some relevant issues surrounding evaluation in 
nonformal environmental education and barriers to effective environmental 
education program evaluation. 

3. To explore strategies for implementing systemic change in the field of environ- 
mental education evaluation and for standardizing the criteria for evaluation of 
environmental education. 

4. To record and document the ideas and strategies explored during the summit 
through the publication of Summit Proceedings. 

Under the guidance of Michael Patton, past president of the American 
Evaluation Association and an experienced evaluation consultant and facilitator, 
the group of invited participants participated in interdisciplinary group discus- 
sions and small group breakout sessions guided the progress of the summit 
toward meeting the above stated objectives. The group of participants shaped 
the course of the Summit as the event unfolded at the Teton Science School, and 
we explored some specific questions including: 
o What is the relationship between evaluation and research? 
o What is evaluation and what purpose does it serve in environmental education 
programming? 

o How do environmental education professionals analyze the costs and benefits 
involved with program evaluation alternatives? 

© What are the standards of practice in environmental education and evaluation 
that would inform guidelines for environmental program evaluation? 

The Teton Summit began a discussion across disciplines that aims to 
improve environmental education and program evaluation in both theory and 
practice. The collaborators and participants at the summit worked together to 
expand the understanding of these disciplines and strategize best practices for 
evaluation in environmental education. 

Society needs quality environmental education programs that adapt to the 
changing needs of learners and educators and incorporate the current socio- 
political context into an ecological framework. Improving upon the ability of the 
environmental education practitioner to incorporate strong evaluations into his 
or her programming benefits the educator and the parent organization as well as 
visitors and students. Not only will such collaboration enable more fundable 
grant proposals, it will illuminate clearer pathways toward achieving the 
established benchmarks for excellence in environmental education. The Teton 
Summit set us down the path toward improving environmental education 
programs across the country. 
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Emmalou Norland and L. Kate Wiltz 



Introduction III 

The field of evaluation has been developing and changing rapidly, and it 
parallels a lot of the issues in environmental education. The changes in both 
fields are driven in part, I think, by the real challenge of this kind of work: that 
of matching response appropriately to need. This involves matching the right 
kind of evaluation to the type of program being looked at and the program’s 
circumstances and needs. It turns out that evaluation is a multifaceted thing, as 
is environmental education. So the challenge ends up being a matching 
challenge, not unlike the challenge in doing good environmental education, that 
is, matching the program to learners’ needs and capabilities. 
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For example, when you are running a wilderness program, the matching 
challenge ends up being finding the right degree of challenge for a particular 
group - not putting them in a dangerous situation, but pushing them enough so 
that they get the fullest experience they can have. You have to make a judgment 
for each group as to what kind of wilderness experience to provide: what trail to 
take them on, how steep and how long, how much weight they can carry. Those 
are the same things you must consider in evaluation - how complicated to make 
the course, how steep it ought to be, how much weight people can carry, how 
long it should go on, and how much risk to expose them to. 

And there are risks with evaluation. Like any powerful tool, evaluation can 
do tremendous good, but unfortunately I’ve spent a lot of time in situations 
where evaluations are doing a lot of harm. Evaluation is not benign. It is not as 
though you either get it right or nothing happens. This can be dangerous stuff 
precisely because it is powerful stuff. Those of you who have experienced the 
down sides know it can affect people’s jobs and livelihoods. It can affect future 
program funding and the higher the program stakes, the higher the evaluation 
stakes. While I look forward to talking about ways of doing evaluation and the 
issues of measurable outcomes, I bring to this discussion the concerns of finding 
myself out there sometimes having to battle with folks who have gone too far 
and are doing harm with evaluation. 

Perhaps the best example of ‘outcomes measurement gone berserk’ is the 
when the body count emerged as the primary criteria for evaluating the progress 
of the Vietnam War. Traditionally, the progress of war was measured by territory 
held, but in a guerilla war ‘who holds territory’ doesn’t make as much difference. 
In Vietnam, territory held in the jungle was an ephemeral notion - and not very 
helpful for assessing the progress of the War. So Secretary of Defense Robert 
MacNamara and his folks, who were systems analysts, decided they needed a 
different measure to monitor how the war was going. They came up with the 
body count. They recognized it as a potentially dangerous measure, so they 
implemented the body count as a strategic measure that only the generals were 
supposed to use. It was not supposed to be used to measure unit fitness. It was 
not supposed to be used at the battalion level, nor at the platoon level. It was a 
strategic indicator of how the overall war was going, 

But what happens when you start measuring things is that, not unlike 
supposedly “controlled” burns in national forests, they can get out of control. 
Sometimes the wind blows from an unexpected direction and things come up. 

On March 16, 1968, when Lt. Joseph Calley went into My Lai and massacred 504 
women, children, and old men, he had just been reprimanded by his command- 
ing officer because his body count was low. He went out to increase his perfor- 
mance on the requisite performance indicators. 

What we are finding across America, which is in an outcomes mania, is that 
a lot of folks are corrupting the indicators. Teachers and principals are cheating 
on standardized achievement tests, not just students, because the stakes for 
teachers and principals in some districts have become very high. The increase in 
high-stakes, accountability-driven outcomes evaluation has led to an increase in 
abuse of evaluation. 



Another consideration is the challenge of doing evaluation in ways that are 
true to our values. It is important that we recognize the complexities of the kind 
of issues we are dealing with, and that we don’t do evaluation in a formulaic way 
or just follow a recipe. Many people do evaluations just to get them done or to 
turn in the paperwork, or just to meet a mandate from somebody. Evaluation, 
through the power of reflection and figuring out what is going on in your 
programs, can support the things that you care about and provide authentic 
evidence that helps you do better those things you’ve dedicated yourselves to. 
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We must look at the power of evaluation techniques both to do good and ill. 

We must look at the values and perspectives that undergird this power. One of 
the most critical venues for doing this is occurring in organizations such as 
those represented by the Summit participants. It is ultimately rewarding to 
work at the leadership level, infusing evaluation thinking into the culture of an 
organization so that evaluation isn't, as it has so often been treated in education, 
an add-on or just extra paperwork. I am always excited when I hear, as with the 
Puget Sound Environmental Learning Center, that they are trying to build 
evaluation in from the beginning of the program planning process because that 
is the way to make it work. Planning for evaluation in this way has not been the 
history of how evaluation has been done and how it came about. But, doing it 
that way and making it real to those involved can become an important and 
innovative part of this environmental education evaluation initiative. ; 

The thing that is challenging about evaluation, when working with people 
who care deeply about what they do, is the skeptical side that evaluation 
represents. It is a skeptical perspective. It is a 'show me' kind of perspective. It j 
doesn't just acquiesce to passion as a good in and of itself. Caring deeply and j 

intending to do good isn't enough. Evaluation, as a skeptical perspective, asks 
for some evidence that passion and caring and the strong beliefs that people 
bring to the things they are doing actually results in the desired good sought. 

In this sense, evaluation infuses reality-testing into our program world of rose- 
colored glasses. 

One of my avocations is collecting data, stories, and research reports about 
what appears to be the nearly infinite capacity of human beings to deceive j 

themselves about reality. The emerging evidence from those who call them- j 

selves neurological anthropologists, who study how our brains develop, is that ; 

one of the strong survival capacities that we have evolved is the incredible ! 

capacity to distort reality. It appears to be a survival capability. When the saber- 
toothed tiger was outside the cave and it looked like everything was hopeless, the 
Neanderthals had this emergent capacity to believe that tomorrow would be 
better than today, despite all evidence to the contrary. We seem to have picked 
up that part of the genome and have carried it forth. The most recent manifesta- 
tion of that, on which massive amounts of research data are emerging, is in the 
stock market. 

One of the largest, most rapidly developing fields, scholastically, is that of 
behavioral finance which is the study of how people make decisions about their 
money. What that field is documenting in very quantitative and very relevant 
ways for those of you managing your portfolios, is that what goes on in behav- 
ioral finance is the same as what goes on in programs. People tend to overesti- 
mate their own financial portfolio gains and program staff tend to overestimate 
their impacts on participants. 

Money Magazine and the Columbia School of Finance did a survey in which 
they asked subscribers to estimate the gains in their portfolios over the last three 
years, and then the researchers actually reviewed brokerage statements to 
determine the real gains in portfolios. Over three-fourths of the sample 
overestimated their gains. A third of the people who said they had made money 
had actually lost money in the stock market. Males overestimated nearly twice as i 
much as females. People tend to remember their winners and forget their losers 
in the stock market — which is exactly what people who run programs do. From 
my experience, the overestimations are nearly the same with respect to the 
degrees of success in programs. 

This environmental education evaluation Summit allows us to bring a 
positive view of reality testing into this kind of work - work that is driven by 
passion, as it ought to be. It allows us to explore how one plays out the tensions j 

between our passions, our beliefs, and the way we would like the world to be and L 
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the evidentiary side of programming: the way that it actually unfolds. These 
tensions emerge both individually within our organizations and collectively in 
the whole field. 

I really think of evaluation, beyond all the methods and beyond all the 
techniques and measurement, as basically the challenge of countering our 
human capacity to distort reality. As a bottom line, evaluation offers a set of 
ways of thinking and tools that can be used to bring some counterbalance to that 
propensity. 

The final challenge that we inevitably confront in gatherings such as the 
Teton Summit is that, while we want to deal with the methodological and 
measurement challenges, we all experience the 80/20 rule in one of its many 
manifestations. In evaluation, that rule describes the phenomenon that most 
people think that 80 percent of evaluation is about methods and measurement 
and 20 percent is about figuring out what to do. I think those numbers are 
actually reversed. Eighty percent of evaluation is about how you think about it, 
and once you get that piece figured out, the methods and the measurement 
actually all fall into place. They’re not that difficult. Issues in evaluation are not 
difficulties of measurement or methods. Those things appear to be difficult 
because of how we think about them. 

At the Summit, we looked a bit at ways of thinking about evaluation (of 
which there are a number) and shared experiences and wisdom around evalua- 
tion and some of the ways the field of environmental education looks at it. I 
think that you will find that as we begin to understand what approaches we want 
to take in evaluating environmental education programs, it will help put the 
methods and measurement piece in context and they will fall into place. The 
traditional way evaluators get themselves into trouble is to begin with the 
methods and the measurement - to come in and start designing instruments as a 
way to begin an evaluation process. So in nonformal environmental education 
program evaluation, we are not going to go there right off the bat. We will sort 
out what the important questions are and look at some ways of thinking about 
that. The Teton Summit has begun that discussion and I hope that it continues 
as we progress into the 21 st century. 

Michael Q. Patton 
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What is Evaluation? 

When delving into a relatively unfamiliar topic or field of interest, it is 
natural for people to make comparisons to that which is already well known. 

They see new ideas, concepts and approaches in terms of more familiar frames of 
reference. Just as when exploring a new culture, some may want to vacation 
there for a week while others may move there to immerse themselves in the 
lifestyle; either person needs to learn and understand the language in order to 
‘get around/ 

The starting point for the Teton Summit, and for this document was explore 
language of nonformal environmental education program evaluation. Agreeing 
on the meanings, at least for this context, of some common terms was key to 
getting around as we explored our topics of interest. First, what is program 
evaluation? 

Program evaluation strives to answer the basic question: how do we know 
what is good? The evaluation process intimidates many people because they see 
it as a passing of judgment or a terminal assessment of good and evil. The 
evaluator comes with magical ways and ties to those with the purse strings. He 
asks questions, pokes and prods, and issues forth findings to be forever cata- 
logued in tomes referenced only by those from whom future funding will be 
sought. The truth is that the evaluator is an expert - but not necessarily an 
expert on the program being evaluated! What then is this expertise that 
evaluators bring to programs? 

Indeed, good program evaluation requires expertise in evaluation. The 
evaluator is proficient in questionnaire writing, surveys, interviewing, report 
writing, to be sure. The expertise involved is less tangible. Program evaluation 
requires an expertise in three primary arenas: 1) Knowing what’s important - 
which distinctions to draw and which to leave; 2) Situation recognition - being 
able to detect, categorize, and place into context a variety of situations; and 3) 
Matching actions to situations in order to achieve that which is really important 
- problem solving. None of these is any small task, but they are not impossible 
either. Program evaluation is meant to inform the program, and when the 
evaluator teams with program experts, the result can do just that. 

Evaluation then, has a context. In this case, that context is nonformal 
environmental education. It is impossible, however, to characterize all 
nonformal environmental education programming with any detail. Some 
commonality does exist among the programs and projects across the country 
that are striving under the guise of nonformal environmental education. 
Environmental education has been variously defined over the past 30 years; 
however one of the earliest definitions has guided practitioners and researchers 
alike: environmental education aims to create an environmentally literate 
citizenry that is knowledgeable concerning the biophysical environment and 
environmental problems, is aware of action strategies for solving those prob- 
lems, and is motivated to work toward their solution (Stapp et al., 1969). The 
Tblisi Declaration in 1977 produced an internationally endorsed view of environ- 
mental education, and incorporated awareness, concern, values, and attitudes, in 
addition to the knowledge, skills and behavior goals for environmental education 
(UNESCO/UNEP, 1978). Today, environmental education continues to incorpo- 



“Program evaluation is the 
systematic collection of 
information about the 
activities , characteristics , 
and outcomes of programs 
to make judgments about 
the program, improve 
program effectiveness , and! 
or inform decisions about 
future programming * 

( Patton , 1997) 
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rate concepts from the traditional disciplines of natural science and social 
studies, including political science, economics, and cultural studies. 



At 

Environmental education 
has been variously defined 
over the past 30 years. . . 



Nonformal environmental education takes place outside the school 
classroom with students of all ages. It includes (but is not limited to) residential 
learning environments, parks, nature centers, zoos and aquaria, and museums. 
Often nonformal learning refers to more than just the physical setting; it also 
implies that learners are actively directing the course of study - that is, the 
nonformal audience is noncaptive. A public program at a local park or zoo often 
attracts an audience of mixed-ages who view their participation in terms of 
recreation and leisure. The school group visiting a park or museum for an 
environmental education program is certainly a captive audience, however; the 
students are not controlling their participation. In this sense, nonformal refers 
to the setting, not the audience. Environmental education is often conducted in 
nonformal settings, incorporates experiential learning techniques and is likely to 
be issue-oriented and place-based. The participants at the Teton Summit 
represented a variety of educational settings and audiences, and the discussions 
on nonformal environmental education evaluation were encompassing of these 
different conceptions. 

The purpose of the Summit on environmental education program evalua- 
tion was not to impose one field upon the other, but to find a common ground 
upon which to explore issues in the evaluation of nonformal programming in 
environmental education. This common ground lies in the better, or sound, 
practices of each field. Better practices, as opposed to the common terminology 
of best practices, is a tacit acknowledgment of the fact that in both fields, there is 
not one best way of doing things. On the contrary, the variety of successful 
environmental education programs and program evaluations are testimony to 
the fact that there are many ways (grounded in theory, research, and practice) of 
achieving our best in both fields. 



At 

Formative evaluation 
Evaluation that focuses on 
program improvement; often 
done during program start- 
up phases in preparation for 
a summative evaluation. 

At 

Summative evaluation 
Evaluation for decision- 
making about the fate or 
direction of a program. It is 
meant to inform funders and 
program leaders as to 
whether or not a program is 
sufficiently successful or 
effective. 




For those of us brave enough to venture into the world of evaluation in the 
spirit of knowing exactly what our programs are doing, we are quickly inundated 
with myriad terms: assessment, summative evaluation, formative evaluation, 
quantitative or qualitative approaches and much more. We must first agree on 
how to use these terms before we can get to know and use evaluation effectively 
in our own programming. This discussion is not a definitive one, however. It 
takes place each time an agency or program begins the formalization of the 
evaluation process. For that is the nature of truly useful program evaluation - it 
is born out of that which is most meaningful to those who are most intimately 
involved in the programming. 

For Whom is Evaluation Done? 

When evaluation is meant to provide information that can be incorporated 
into current programming, it is often considered ‘formative.’ Formative 
evaluation is something we often do every day informally as we observe partici- 
pant reactions, test out a new activity, or count the number of participants in 
our program. Formal evaluation that informs program improvement is not 
meant to provide information for decision-makers on funding or to assess the 
program’s progress toward goals; it is meant to guide the subtle changes - such 
as solving unanticipated problems or checking participants’ progress toward 
goals - that can make a program better meet its goals. 

Summative evaluation, on the other hand, is evaluation done to determine 
the relative ‘success’ or merit of a program. As such, it is typically done toward 
the end of a program’s life or funding cycle. Summative evaluation reports on a 
program to an “external” audience such as funders or legislators or future 
patrons who will pass judgment on the program (Scriven, 1994 describes 
summative evaluation in detail). The reality of summative evaluations is that 
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they are often just historical documents, as they are performed months after the 
decision to re-fund, expand or cut a program has been made. Therefore, since 
formative information is on hand when these decisions are being made (often a 
year before the program’s development cycle is completed), formative data are 
often used, inappropriately, in a summative manner. 



What other words are used by environmental educators and evaluators that 
may need some clarification? Assessment is one. Evaluators often use assess- 
ment to refer to the testing, often cognitive, of individuals, for use in judging 
individual performance. This is in contrast to the broader term of evaluation, in 
which individuals may be tested, but the data are analyzed and applied on a 
collective level. The data are used to judge a program’s performance. In 
environmental education, individual student assessments can inform a program 
evaluation, particularly if a primary desired outcome of programming is student 
learning. In nonformal settings, however, student assessments are not always 
feasible and may not be the most practical way to gather data about program 
participants. 

Outcome is another term with complicated connotations. Outputs, 
outcomes, and impacts all refer to those results of our programming that we 
hope to achieve and measure, or at least document, with evaluation. While 
outputs are typically numbers (as in participants) or products (as in books or 
courses), outcomes are generally broader concepts, (reactions, affective or 
cognitive changes) that the program participants take away from the program 
experience. To inform environmental education programming, both outputs and 
outcomes must be measured. Knowing a program’s participant numbers and 
products may not always be the most useful information to the program 
developers. Accounting for participant reactions and their attitude and behavior 
changes after a program is often much more meaningful to those directing the 
course of an environmental education program. Program impacts are longer- 
term outcomes, and include those secondary outcomes in which groups beyond 
those who participated in the programming are affected or change as a result of 
the program. In environmental education programming, our goals are often 
stated in terms of broad program impacts such as stewardship for a resource or a 
more informed citizenry. 




Program evaluation is undertaken for a variety of reasons and with a variety 
of anticipated benefits. Often in nonformal environmental education, we are 
seeking to affect the participants’ attitudes or feelings about a place or a 
resource. Alternatively, we sometimes want participants to feel empowered to 
make decisions and act on particular issues. These are very difficult attributes to 
measure. However, at some point in the program process, it becomes important 
to verify whether participants’ attitudes, feelings, and behaviors are changing - 
whether the program is achieving its stated purpose. Programmers want to 
know whether their programs are working. Funders want to know if their 
money was well spent. 

Evaluation should be useful - it should answer the questions that those 
involved need and want to have answered. Leaders in educational organizations 
have to be able to identify their program weaknesses in order to learn and to 
improve programming. Evaluation provides a basis for program improvement 
that goes beyond cursory feedback that often says, “Everything is going great!” 
Program evaluation can target specific valued outcomes and provide the 
information program leaders need and want. 

In most fields, a supporting research base lends context to the practice, and 
program evaluation questions are often easily derived from a combination of 
specific program outcomes and the implications indicated by current research. 






Developmental evaluation 
Evaluation undertaken for 
the purpose of supporting 
program, project, staff, or 
organizational 

development. The 
evaluator teams with 
program leaders to 
facilitate long-term 
discussions and 
explorations of continuous 
program improvement, 
adaptation, and intentional 
change. 



“It is often better to have weak 
measurements of important 
things than to have strong 
measurements of unimportant 
things. ” 

Michael Q. Patton 



Evaluation should be useful - 
it should answer the questions 
that those involved need and 
want to have answered. 
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In nonformal environmental education, the research base is relatively undevel- 
oped and the program outcomes are often difficult to measure and nearly 
impossible to connect to long-term impacts. Connecting a student’s participation 
in a wilderness program to a subsequent voting record on environmental issues is 
quite a task. Program leaders tend to determine the criteria for evaluation based 
on the desired outcomes that they value and see as most pertinent to furthering 
their educational mission. Without research to strongly correlate particular 
outcomes such as ‘positive attitude toward polar bears’ to larger mission goals 
such as ‘stewardship of the arctic nature preserve,’ program leaders must rely on 
their mission, experience, intuition, and/or instincts and values to guide their 
program evaluation questions. 

As research is generated in nonformal environmental education, the linkages 
will become more clear, and the variables indicating a program’s success will be 
more easily measured. Until then, individual programs will have to determine 
their own criteria, drawing from experience, existing research, and other evalua- 
tion data. As a discipline, too, nonformal environmental education will continue 
to clarify the common concerns in evaluating environmental education programs. 

We must be wary, though, that we don’t fall into the trap of measuring that 
which is easy to measure, deriving outcomes from those results, and letting those 
outcomes determine what it is that we value in our programs. This process is 
backwards. Measurement follows from clearly understanding what the outcomes 
should be - outcomes that are determined by values, such as valuing stewardship 
behavior or valuing critical thinking skills. Instead of allowing measurement to 
drive outcomes to determine a sense of program value, good evaluation of 
environmental education will have that which we value in our programs driving 
our desired outcomes, and those outcomes determining measurement. It may 
not be easy to measure the stewardship ethic in program participants, but if that 
truly is what the program aims to instill, and it is what the stakeholders of a 
program want to know about, it is the variable that must be examined. 

Even when environmental education programmers are agreed as to their 
evaluation questions and desired outcomes, other issues can arise that complicate 
the practice and use of evaluation. How we choose to operationalize our evalua- 
tion questions and the outcomes to which they refer can result in differences in 
professional opinion as to the best evidence or methodology to use. 

If a student who experiences an environmental program with an ultimate 
desired outcome of informed environmental decision-making chooses to maxi- 
mize his or her energy consumption instead of conserving, have we failed to meet 
our program goals? Is the student applying his knowledge in a thoughtful and 
responsible way, or has the message of informed decision-making been lost on 
this student? How do we know that the participant has met our goals? When 
these types of value-laden disagreements exist, often what is needed is an infusion 
of data into the philosophical debate. Although in the long term, research in 
environmental education will help guide our programming, evaluation can 
provide individual environmental education programs with the data they need to 
improve today. 



How is Evaluation Done? 



The Joint Committee on Standards for Educational Evaluation adopted a set 
of standards for the practice of evaluation in 1981 and revised those standards in 
1994. These are the guiding principles for better practices in the field of evalua- 
tion, and are at the forefront of good program evaluation in nonformal environ- 
mental education. There are four primary categories of standards. They are: 
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Utility 

The Utility Standards are intended to ensure that an evaluation will serve the 
practical information needs of intended users. 



Feasibility 

The Feasibility Standards are intended to ensure that an evaluation will be 
realistic, prudent, diplomatic, and frugal. 

Propriety 

The Propriety Standards are intended to ensure that an evaluation will be 
conducted legally, ethically, and with due regard for the welfare for those involved in 
the evaluation, as well as those affected by its results. 



Regardless of the theoretical 
stance or methodology, the 
evaluation standards should 
guide the practice of 
evaluation. 



Accuracy 

The Accuracy Standards are intended to ensure that an evaluation will reveal 
and convey technically adequate information about the features that determine 
worth or merit of the program being evaluated 

Regardless of the theoretical stance an evaluator takes or the preferred methods ; 
employed, the above four standards for evaluation should guide practice. In fact, 
program evaluators do come with a variety of approaches and methodologies - some 
prefer participant questionnaires, others pour over the documents and materials 
behind the program, while still others rely primarily on direct observation or 
intensive interviews to gather the desired information. In addition to different 
methodologies, evaluators also subscribe to different approaches and philosophies as ] 
to what should be measured. Some look closely at the stated program goals and 
objectives to guide the evaluation questions. Others determine evaluation questions ! 
based on the concerns and questions of stakeholders involved with the program. 

Often a process referred to as logic modeling is used to help map out the pieces j 
of an organization’s program and provide a mutually understood framework for j 

understanding the intent and functioning of the program. Logic modeling links all | 

of the inputs to a program (money, supplies, human resources, participants) to the j 
desired outcomes of that program (reactions, attitudes, knowledge, behaviors). j 

Bennett’s Hierarchy (Bennett and Rockwell, 2000) is a useful model for mapping out 
a program logic (see Figure 1). In stakeholder-based evaluation, often the evaluator 
will work with the program leaders to clarify the program components that ‘fill in’ 
the levels of the hierarchy. For young or very broad programs, this can be an 
engaging learning experience to see how the activities that they do with individual » 
program participants directly - or indirectly - link to program outcomes. 
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I The logic modeling process is 
i ' most effective if there is 
research that, supports the 
connections between the levels 
of the hierarchical model 




The logic modeling process is most effective if there is research that supports 
the connections between the levels of the hierarchical model. Educational 
research can often provide the projections that connect activities to specific long- 
term outcomes and program impacts. It is rarely possible to follow program 
participants for years after their environmental education experience in order to 
determine if their hands-on learning activities in woods contributed to positive 
generalized environmental attitudes expressed in the voting booth. Therefore, 
environmental educators and evaluators must rely on established research to 
support these connections, while they focus their evaluation efforts on answering 
other important, yet more feasibly answerable, questions. 

So do we throw out all hopes of aspiring to rigorous measurement, logic 
modeling, and proving our programs impact under the cloak of feasibility? No. 
Indeed, accuracy is another of the evaluator’s credos. Evaluation requires some 
level of proficiency in applying valid and reliable measurement techniques, often 
taken directly from the research journals, to the study of evaluation questions. 

One difference between research in nonformal environmental education and 
evaluation of nonformal environmental education programs lies in the purpose 
behind each. Although it is true that often research is done in the context of 
evaluation projects and vice versa, program evaluation is driven by utility - it’s 
primary purpose is to produce information that is used by the program leaders. 
Alternatively, research tends to be more concerned with contributing to a body of 
knowledge. 

In environmental education, as with many disciplines, funding is tight and 
often when there is an opportunity to evaluate a program it is merged with 
research meant to inform and advance the field. Often it seems that research and 
evaluation projects race neck and neck in the competition for funding. This 
situation begs the question: what are the appropriate agendas for environmental 
education research and evaluation? 

During the Summit, participants worked in small groups to explore possible 
lines of inquiry for future research in nonformal environmental education as well 
as possible evaluation questions that applied to many of our programs. Many of 
the themes that emerged in the discussions have both evaluation and research 
implications and could be found, addressed at different scales, on either agenda. 
Here are some of the specific topics and questions participants suggested: 

Outcomes and affect 

0 What is environmental sensitivity? - knowledge, human-nature, connection, or 
affective/emotional experience? 

• What are the affective outcomes of environmental education programming (e.g. 
citizenship, civic responsibility, community participation, values, spirituality)? 

• How do experiences affect stewardship behavior? 

® What environmental education methods are related to which outcomes (e.g. 
residential learning)? 

0 How does the role of gender affect the outcomes of environmental education? 
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Organizational issues 

• What are some challenges to implementing environmental education? 

0 Does environmental education relate to the goals and missions of the larger 
natural resource agency or natural resource management? 

® Is the mission of the environmental education agency actually communicated 
in the programming? 

• How is the way in which program planners and implementers are involved in 
the design of evaluation influence the use of evaluation results? 

• What is the level of organizational support for program evaluation? Is there staff 
‘buy in? 7 
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Approaches to environmental education 

© Does the length of an environmental education program matter? 

© What part(s) of our programs are making the differences? 

© What are the goals of “good” environmental education? 

© What is the difference between advocacy and educational programs’ impact 
on kids? 

© Does participation in short term programs lead to participation in longer- 
term programs? 

© Where should we spend our time? - knowledge experience, affective/sensitiv- 
ity experience, emotional development/reactions? 

© Is there a difference in outcomes between family programs and youth 
programs? 

© Are problem-solving skills enough without place-based skills? 

Family and lifestyle 

© How is current lifestyle (with relatively fewer outdoor experiences) affecting 
kids? 

0 What do parents think about environmental education programs? 

0 What do parents learn from their kids? Is there a trickle-down effect? 

© Does mentoring of younger kids by older kids in environmental education 
have effects on self-esteem? 

Longitudinal outcomes and impacts 

© What are the long-term effects of environmental education? Longitudinal 
data are needed to understand our impacts; however, the benefit is sometimes 
exaggerated. Long-term data following a period of 3 to 5 years may be the 
most beneficial, but longer studies can garner diminishing returns as the 
popular culture changes and attitudes begin to reflect those changes. 

° Does environmental education make a difference? 

Relationship to/role in formal education 

© What are the benefits of outdoor experiences or residential experiences as 
compared to classroom science? Are there topics that are better learned 
outdoors? 

© Is our environmental education relevant to state standards / curriculum 
development? 

© How do environmental education programs relate to formal pedagogical 
programs? 

© Teacher training - impacts of long-term professional development, influence 
on students? What teaching techniques do teachers gain from our profes- 
sional development programs? Do they use them? How does exposure to 
environmental education affect teachers? 




Research and data collection 

© Are anecdotal data useful, valuable? What is the best way to use them? 

• How does neurological research apply to environmental education? 

• Compilation and synthesis of research data from the fields of environmental 
education, adult education, science education, psychology, and natural 
resources? 

• Can we do an experimental designed study on environmental education 
participants vs. non-participants 

• What does the environmental education audience look like in terms of their 
expectations and abilities? 

Role of technology 

• What is the difference between virtual and real experiences in terms of 
efficacy? 



fa 

Program evaluation is the 
localized, quality control 
department of the research 



world. 






15 



mm 



m. 



i£ 



t^t 



SSKH'lig 



ik L 



EW” 



irPT 



X* ,j 

^ L. 



0 What is “appropriate” use of technology? What are the issues of age appropri- 
ateness, optimal curriculum? 

What is and What Should Be 
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Both research and 
evaluation generate 
knowledge. What kind of 
knowledge, the depth of that 
knowledge, and the intended 
used of that knowledge can 
differ tremendously. 



When a field is well defined and established, research - coupled with 
practical experience - can drive the practice of a discipline. Standards of practice 

- at least in the form of minimum standards - are determined by research 
findings. The research agenda for many fields tends to establish the core 
elements of the practitioner’s model - what it takes to constitute an ‘interven- 
tion/ to use research terminology. In fields such as clinical psychology, the 
scientist practitioner approach calls for each program to act as a mini research 
case study. The psychologist is encouraged to actively pursue a research agenda 

- collecting data via systematic observation - while practicing the trade. In 
environmental education, the front-line educators or interpreters often don’t feel 
they have the time or expertise to formally advance research in the field. The 
questions that they have, though directly related to those of researchers in the 
field, are driven by their desire to maintain or improve their own programming 
or maximize their efficiency at delivering the educational or interpretive 
product. 

Program evaluation is the localized, quality control department of the 
research world. It has inherited the classical forms of research terminology - 
‘dosage’ refers to the length of a program; ‘side effects’ of programming include 
the risks to participants; and ‘treatment fidelity’ or purity refers to the models 
that we use in our approach to programming: are our environmental education 
programs place-based or issue-based? Do we rely solely on hands-on activities in 
the outdoors, or do virtual activities augment our program? Do we structure 
our program as part of a long-term set of activities for participants or is it a 
stand-alone experience? How we structure our programming depends on a lot of 
factors, not the least of which is practical experience as to ‘what works' and the 
demand that our clients express for particular services. Evaluation can connect 
those driving forces to the environmental education theory and research that 
supports various approaches to programming. Ideally then, program evaluation 
functions in a quality enhancement capacity. Research in the many disciplines 
that inform environmental education - social psychology, teaching and learning, 
science education, and adult education - informs us as to the foundational 
guidelines for programming. 

Although there is some discussion as to how well developed these functional 
guidelines are, they exist. In environmental education, these guidelines may 
only refer to safety standards and minimum competencies - and as long as they 
are viewed as minimums and don’t become the guiding force in what we do, we 
have room for the quality enhancement that evaluation offers. Whether specific 
programs are framed by intrinsic arguments or carefully crafted from the 
standards, there is room for program improvement via evaluation linked to 
recent and relevant research. The National Park Service has taken a step beyond 
the minimum standards approach with their guidelines for interpreters. This set 
of documents uses a common language to bridge interpretation and education in 
the Park Service and describe what it is that interpreters do and what it looks 
like when it is done well. Other documents exist that guide the environmental 
education and interpretive professions (see recommended readings, Appendix B). 



Evaluation and research differ in scope and purpose. Evaluation is setting 
specific, and often has less depth and generalizability in its instrumentation. 
Evaluators may use instruments with shallow and specific characteristics, to 
answer specific questions rather than track underlying trends. It is a part of the 
research agenda to collect data that are more profound and generalizable across 
programs. As described earlier, evaluation is primarily done for making deci- 
sions about a program. The use of research findings, however, is typically more 
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focused on fundamental judgments and/or decisions regarding the value or 
worth of the program or model being tested. These differences may be subtle in 
the case of large-scale evaluations where findings are published and shared to 
inform similar programs; however the differences in purpose between evaluation 
and research most often mean that they ask different questions which indicate 
different methods and produce different findings. Both generate knowledge. Of 
that there is no question. What kind of knowledge, the depth of that knowledge, 
and the intended uses of that knowledge can differ tremendously. 



The P©flitDCS ©f E¥alyatD©sn 

Evaluation generates knowledge, and knowledge is power. And, perhaps 
more important to agencies with an educational mission, money follows power. 

It is important to recognize and acknowledge these facts. Whoever determines 
the evaluation questions and agendas holds the key to what information will be 
gathered. On a program by program basis, these decisions can impact large 
numbers of educators, administrators, funders, and clients or visitors. Across 
the environmental education and interpretation fields, these evaluation ques- 
tions (often mixed with research questions) are driving the goals and objectives 
of practice. 

Although participatory evaluation and stakeholder-based approaches put the 
decision making power in the hands of those affected by the evaluation, evalua- 
tors hold the key to making an evaluation happen. They establish the evaluation 
protocol and determine whether a participatory process will or will not occur. 
They offer the menu of approaches, techniques, theories, and methodologies. 
They often collect the data and they write the evaluation reports. 

Formalizing the profession of evaluation, or any profession, invites 
standards of practice, ethical guidelines, and often results in the credentialing of 
practitioners. Credentialing or licensure is one way to acknowledge, and 
legitimize, the power that comes with the responsibilities of working within a 
profession. This final step has yet to completely take place in both the fields of 
evaluation and nonformal environmental education. However, there are current 
discussions among members of the professional organizations regarding the 
need for and the requirements of installing a certification process. In 1998, the 
National Association for Interpretation introduced its certification program for 
heritage interpretation, and has three other categories for certification as well. 
The American Evaluation Association, although it does not certify professional 
evaluators, endorses a training program and offers many professional develop- 
ment opportunities. These concepts - credentialing, licensure, certification - 
may not be necessary for quality programming, but they shape a field’s common 
language, help define what it is that we do, and help us explain our professions 
to others. 



What EowiroomeBital Education) 
Eva!yati©m Cam L©©k like 

The Summit was organized and partially funded by two primary partners in 
an ongoing environmental education collaborative. Emmalou Norland directed 
the evaluation of the project that was managed by Julia Washburn of the 
National Park Foundation. The two of them were asked to discuss their project 
in terms of the lessons they have learned and paths they have traveled in 
coordinating the evaluation of a very large and complicated environmental 
education grant. 

The Parks As Resources for Knowledge in Science (P.A.R.K.S.) program is a 
nationwide environmental education grant managed by the National Park 
Foundation and funded by the ExxonMobil Corporation. Thirty-six National 
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Evaluation is not political when 
it meets the following criteria: 

°No one knows about it 
, ®No one cares about it 
(>There is no money at stake 
' °There is no power at stake 
I °No one is making any decisions 
j based on it 

| °No one participates in it 

I 
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Michael Patton 
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Park sites were chosen to receive funding to infuse the National Science 
Education Standards (NSES) into new or existing park programming for school 
groups. The National Park Foundation partnered with the National Science 
Teachers Association (NSTA) and the Ohio State University, while individual park 
sites formed local partnerships with schools and other community organiza- 
tions. Each local P.A.R.K.S. team created a project plan that focused on using 
their park resources and serving local school groups by incorporating the NSES 
into the park curricula. Although the audience for P.A.R.K.S. programs rests 
firmly in the realm of formal education (school groups across the country), the 
projects’ settings were typically nonformal (that is, National Park sites) with a 
few programs conducted in the school classrooms. 

Program topics ranged from rice farming in South Carolina, geology in 
California, water quality in Texas, Lewis and Clark’s expedition in Oregon, to 
marine studies in Alaska. Program participants were also quite diverse: some 
programs served elementary students, some middle schoolers, some high school 
students, and some programs were directed toward teachers themselves. With 
the wide range of program foci, locations, and participant ages, the P.A.R.K.S. 
program evaluation was quite large in scope. At the local level, for example, 
projects could benefit from the incorporation of formative evaluation into their 
program. With that in mind, parks were encouraged to form stakeholder groups 
representing their local partnerships and those with an interest in the outcomes 
of their P.A.R.K.S. program. Three regional workshops were held early in the 
grant period in order to familiarize the 36 teams with the NSES as well as 
evaluation processes that could be incorporated into their program planning. 

The workshops were conducted by NSTA and the Ohio State University evalua- 
tion team. 

On a national level, a summative evaluation was needed in order to meet the 
needs and questions of the P.A.R.K.S. grant partners. In order to keep the focus 
of the evaluation on the overall program, a cluster evaluation format was 
chosen. Data would be collected from the individual park projects, but all 
analysis and reporting would be aggregated. 

A stakeholder-based evaluation approach was implemented in which 
representatives from the National Park Foundation, ExxonMobil, the National 
Park Service, and NSTA were selected to participate in crafting the questions 
that the evaluation would address. Data collection consisted of site visits to the 
each of the park sites during which the local park-teacher teams were inter- 
viewed, programs were observed, and the park resources (physical and cultural 
contexts) were assessed. Additionally, questionnaires were sent to the teachers 
and students who participated in the parks’ programs. 

With such a large project and many individual parks creating their own 
evaluation plans for program improvement, the road was not always smooth 
during the course of the grant period. The evaluations (both local and national) 
had to be adaptive and reactive to the inevitable changes in program plans, hold- 
ups in implementation, and other snags as the parks began their work. 

RA.R.K.S. as a IVtodeS 

So, what aspects of the P.A.R.K.S. program evaluation can be modeled 
effectively by other nonformal environmental educators? There are several 
identifiable strengths in the P.AR.K.S. program and evaluation approach. First, 
the required collaboration on both the local and national level was an integral 
part of the initial vision of the P.A.R.K.S. grant. Collaboration among multiple 
partners contributing to a project often improves the chances for that project’s 
sustainability beyond the life of the grant. In order to determine if that is the 
case with the P.A.R.K.S. grant, the national evaluation team researched organiza- 
tional collaboration and measured the extent and quality of the local teams’ 
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collaborative efforts. This approach to measuring the variables that answer 
stakeholders’ evaluation questions is the second strength of the P.A.R.K.S. 
evaluation. Grounding the approach to measuring such variables as collabora- 
tion and stewardship in the relevant current research literature allows this 
project to further the knowledge in these fields of inquiry while gathering 
information that is useful to the stakeholders as well. 



Another strength of the P.A.R.K.S. program is its flexibility. From the 
beginning of the grant’s implementation, the evaluation plan changed and grew 
to account for the changing characteristics and needs of the stakeholders. Some 
National Park educators transferred to other parks, a budget cut removed some 
team teachers from their jobs, a drought prevented the renovated rice fields from 
maintaining the proper water level, and on the national level Exxon merged with 
Mobil and the funder representative to the stakeholder team was replaced. 
Fortunately, recognizing inevitability of change over course of grant was a lesson 
that was quickly learned. The national partners were open to creative solutions 
to handling unexpected changes at all levels of the project. Adaptive, reactive 
evaluation management is key to making sure that the evaluation continues to 
yield useful, relevant information for those who need it - even when those people 
are not the same who started with the project. 

The final key to the success of the P.A.R.K.S. project evaluation is that from 
the initial phases of planning the project, money was designated for the evalua- 
tion. With the P.A.R.K.S project, both Exxon and the National Park Foundation 
recognized the benefits of funding a national evaluation of their program, and 
they incorporated a formal evaluation component into the grant from the 
beginning. Other funders of environmental education in both formal and 
nonformal settings are asking for evaluation plans in all of their program 
proposals. Funders have a vested interest in seeing sustainable programs, and 
evaluation can help programs succeed by redirecting efforts in an evolving 
manner. 



As the National Park Foundation and the Ohio State University have both 
learned, another necessity for effective evaluation is building in enough time. 
When attempting to put in place the capacity for ongoing program evaluation, 
one must recognize that it takes time to evaluate. Evaluation does not just 
happen on the side. Environmental educators need to set aside time to address 
their evaluation questions, meet with stakeholders if that is their preferred 
approach, and plan their data collection. 



The P.A.R.K.S. model for successful evaluation: 

1) Build in time and money for evaluation 

2) Adapt and react - plan to change the plan 

3) Recognize opportunities to connect evaluation to research 

4) Encourage collaboration and other opportunities for sustaining the program 

Valuing Programs from an Economic 
Perspective: Cost Analysis in 
Environmental Education Evaluation 

Although evaluators typically are not trained in cost analysis, those 
requesting and/or funding an evaluation often call for this skill. The costs 
associated with environmental education programs and the benefits those 
programs provide to participants, local communities, or society in general are 
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often key variables in a formal evaluation. The way in which this cost analysis is 
approached can have dramatic effects on the results. Therefore, it is important 
to conceptualize carefully the role of cost analysis in program evaluation. 

Cost analysis links dollars to numbers of participants or instructors or other 
line items in the list of program components. Understanding the environmental 
education program, its context, and seeing the program inputs and outcomes as 
they relate to each other can be key in uncovering true program costs and 
effectiveness. As mentioned before, the process of linking program inputs to 
outcomes though logic modeling can aid in seeing the overall picture of a 
program. It also can aid in identifying the discrete components of a program - a 
process necessary in order to begin analyzing program costs. In the end, cost 
analysis is the process of translating a program’s budget into a program-based 
language. 

Cost-benefit and Cost-effectiveness: 

As professionals in the field of environmental education program evalua- 
tion, we are interested in more than just the costs of different programs - we also 
need to account for the outcomes, or benefits of our programs. Cost-benefit 
analysis brings in the outcome evaluation orientation and begins to link dollars 
to the outcomes of programming, not just to program activities or inputs. Cost- 
benefit analysis assigns monetary value to all components of a program - inputs, 
outputs, outcomes, and societal impacts. By taking the varying pieces of a 
program and reducing them to dollar terms, the process works particularly well 
when considering programs with widely varying outcomes that would otherwise 
be difficult to compare. 

The cost-benefit analysis of a program requires that a program be broken 
down into its constituent parts, and can work quite effectively with the logic 
modeling process described earlier. Cost-benefit analysis uses data and current 
research to link evident program outcomes to the longer-term societal benefits 
that are harder to measure. When looking to justify the field of nonformal 
environmental education, cost-benefit analysis can put environmental education 
program outcomes in terms more easily compared to those of other social 
programs. 

Another form of analyzing costs and program outcomes is cost-effectiveness 
analysis. This process is inherently comparative and is often more appropriate 
when the alternative program choices being considered are not drastically 
different. For instance, the decision may be choosing between an outdoor 
wilderness program that incorporates a ropes course versus one that uses 
science fieldwork instead. The desired outcomes may be similar enough among 
a particular agency’s programming options that effectiveness analysis is more 
appropriate. Unlike, cost-benefit analysis, cost-effectiveness analysis does not 
convert program outcomes to dollar figures. Although the process still renders a 
ratio of cost to benefit, the benefit, or effectiveness, quotient is thought of in 
terms that are more inherent to its nature. If the desired outcome of a program 
is that participants go on to finish high school or to major in science in a post- 
secondary program, measures of this outcome (number of graduates per 
program, etc.) may be the simplest way to compare the program alternatives. 

The cost-effectiveness approach will typically be the easier and more 
accurate approach when environmental education programmers are trying to 
make decisions regarding alternative approaches or methods in their programs. 
In environmental education programming we are often looking at program 
alternatives that are quite similar. Additionally, environmental education 
programs often aspire to societal or environmental impacts that are difficult to 
characterize in terms of dollar value. Cost-effectiveness analysis addresses these 
difficulties by generalizing the costs of program inputs and outcomes over time. 
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It must be a researched process - not a guess. The process of cost-effectiveness 
analysis requires the use of existing research data to assign value to program 
outcomes. 



The process of cost effectiveness and cost benefit analysis is inherently 
comparative and results in a decision-making process that has a long history in 
politics and business. Program A and program B are compared both in terms of 
cost and benefit (see Figure 2). Typically a clear advantage on both scales is not 
the case for one program. More often, a program will rank highly on one scale 
and lower on the other, making the choice less obvious. In this case, cost 
effectiveness analysis can clarify the options. If program A costs more than B, 
and it appears that program A accomplishes more of its desired outcomes, one 
must analyze the effectiveness of the programs further. Instead of deferring to 
the cost argument (in which program B would be less expensive’, research on 
the societal value placed on the programs’ outcomes and impacts can be 
employed. A dollar value can be extrapolated from logically connected program 
outcomes and impacts. These outcomes and impacts can then be more carefully 
scrutinized to determine which program is the better value. 



Figure 2: Cost-Effectiveness Analysis 
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Other Factors when 
Considering Costs: 

With respect to program inputs, cost-benefit and cost-effectiveness analysis 
can bring out the value of a program’s resources - a process that is complicated 
by the variety of less apparent resources involved in many environmental 
education programs. In addition to the real costs associated with things like 
supplies or travel, in-kind contributions such as operating costs or time 
contributed by volunteers often undergird larger programs. These resources are 
critical to the success of any environmental education program - in experiential 
or wilderness programs, the value of the land set aside for program use is 
another additional resource. These resources can serve to leverage funding, and 
a program’s sustainability may rely heavily on the understanding of these less 
apparent resources that are not easily documented in line-item budgets. 

In order to ascertain the costs associated with any program, care must be 
taken to break down that analysis into those costs associated with the start-up of 
a program, the core costs needed to maintain the program, and the costs of the 
‘bells and whistles’ that enhance programming. Start-up costs, of course, inflate 
the overall program budget and when identified as such can be reallocated in 
future budgeting. Carefully and accurately identifying the core elements of a 
program can be critical when an evaluation is helping practitioners to make 
decisions about the sustainability of a program. In addition, understanding the 
core of one’s program helps to insure against stripping those elements in the 
name of cost-saving or sustainability at a later date. 
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Additionally, determining if there is an economy of scale, in which the cost of 
an individual program decreases each time that program is given, is imperative in 
understanding the costs associated with changing the program or modeling it 
elsewhere. Finally, those costs identified as ‘unusual’ or stemming from unex- 
pected crises are often factored out of the core program budget reports. However, 
particularly if they are repeated across projects or budget cycles, unusual costs 
must be monitored, estimated, and accounted for in a cost analysis. In other 
words, we must learn to expect the unexpected costs. 



Should Environmental Education 
Program Evaluation be Required? 



Wildlife watching: Cindy Somers, 
Stephanie Pittard, and Kate Wiltz 




More and more funding agencies, in an effort to encourage effective program- 
ming, are mandating program evaluation in their funded proposals. Grantees are 
being required to include evaluation components as a part of their program plan in 
order to obtain funding for the program itself. Obviously, incorporating the 
evaluation piece affects the cost of programming. Program leaders often want to 
understand how they can justify the cost of formal program evaluation. Avery 
general rule regarding funding environmental education evaluations encourages 
that 10 to 15% of the total project budget be devoted to its evaluation. 

In situations such as the P.A.R.K.S. project discussed earlier, where the funder 
recognizes the additional expense that evaluation entails and the program leaders 
are interested in and enthusiastic about the evaluation, this model can produce 
successful programs. However, forcing program evaluation on environmental 
education programs can cause problems. In some cases, where the evaluation is 
not fully appreciated by the programmers, its implementation can interfere with 
the program itself. Additionally, if the program leaders are not interested, the 
possibility of them using the results of the evaluation is considerably diminished. 
The issue of mandated evaluation components in environmental education 
programs should be carefully considered, and quite possibly program evaluation 
will best serve those who are more ready and willing to welcome it into their 
programs. 

Possibilities: Where D© We 
Go From Here? 

In the corporate sector, organizations have research and development (R & D) 
arms that allow companies to continually improve their products and meet 
consumer needs and expectations. R & D departments incorporate marketing and 
production research with the specific theoretical and science-based research 
regarding the products that they create. The entire process of product design, 
production, distribution, marketing, and consumer use and satisfaction is evalu- 
ated, and the information gathered is used to improve this process. Corporate R & 
D basically consists of targeted research performed on very specific subjects that 
will inform decision-making within the organization. 

Sound familiar? The nonprofit world does not have a formalized R & D sector; 
but we do have formal evaluation that serves the same purpose for our programs. 
Program evaluation is the research and development arm of nonprofit and 
governmental educational programming. It is tailored to a particular program or 
agency and is designed to aid in informed decision-making for program improve- 
ment or continuation. When thought of in this way, the importance of program 
evaluation begins to emerge. Its role as a natural step in program development 
and implementation in nonformal environmental education becomes clear. 
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At the end of the journey to the Summit, we find that nonformal environ- 
mental education needs effective, useful evaluation in order to enable the field to 
grow and develop in our changing society. Program evaluation must be 
adaptable to the demands of outdoor, experiential education with affective 
outcomes and longer-term goals associated with relatively brief program 
'dosages/ Most importantly, we find that the process of sharing what has 
worked, what didn’t, and how to use evaluation information is perhaps the key to 
improving nonformal environmental education. Communication among 
practitioners, funders, and the academicians working in environmental educa- 
tion is imperative to creating programs that meet the needs of the target 
audiences while designing evaluations that meet the needs of those involved in 
programming. Following are some ideas, suggested by the participants at the 
Teton Summit, for fostering such communication. 

A first step is the creation of affinity groups that sponsor environmental 
education evaluation training opportunities and organize events and presenta- 
tions at national conferences in both nonformal environmental education and 
program evaluation. Venues for activities include the professional associations 
of which many of the participants are members: the North American Association 
for Environmental Education, the National Association for Interpretation, the 
Visitor Studies Association, the Association of Nature Center Administrators, and 
the American Evaluation Association. The Environmental Program Evaluation 
Topical Interest Group of the American Evaluation Association, begun in 2000, is 
an example of such an affinity group. 

In order to reach a broader audience with evaluation information, any 
training workshops delivered to nonformal environmental educators should be 
designed from a multi-level, multi-focus approach in which educators can easily 
obtain information that is relevant to their content area and program type. The 
creation of a guidebook or set of guidebooks may be the best way to encapsulate 
the information in an organized and useful way for nonformal educators. Such a 
handbook might consist of: 
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® A list of environmental education outcomes 

• A current research agenda for environmental education 

• Better practices for environmental education evaluation 

© A 'how to’ section for including evaluation in environmental education 
programs, including consultation information and contacts for professional 
evaluators 

0 A cross-referenced list of resources on program evaluation for environmental , 
education and nonformal environmental education 
© A collection of lessons learned from evaluations in nonformal environmental 
education settings 

0 Examples of quantitative and qualitative evaluation instruments that have 
been utilized in nonformal environmental education settings, including 
sample permission forms, cover letters, questionnaire packets and incentives 

Of course, this information can be compiled and shared in ways other than 
the publication of a guidebook. Simply by incorporating the funders and 
practitioners in nonformal environmental education into the processes tradi- 
tionally reserved for academia, the information gap between theory and practice ; 
can be bridged in both the evaluation and environmental education fields. 

Program leaders who are developing evaluation components can utilize peer 

review for feedback on their program evaluation. Other environmental educa- • 

tors and program funders, as well as researchers can review instruments or act 

as stakeholders to the evaluation team. Alternatively, those who conduct ' 

programs can play a role in forwarding research projects in the environmental j 

education field through a similar process of reviewing research proposals and : 

article submissions. Ideally the overall goals for each group are the same: better 

practice of nonformal environmental education. The lines between theory and L. 
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practice are often blurred and somewhat arbitrary. A stronger front can be 
presented in advancing the field of nonformal environmental education evalua- 
tion when practitioners, researchers, and funders mutually support its growth. 




As one approaches the National Museum of Wildlife Art along the road 
north of Jackson, only subtle changes in the landscape seem to hint at the 
museum’s presence. From a distance, the charred remains of log poles can be 
seen, erected straight and tall, against the sage covered hillside. These sentinels 
draw the eye upward, and slowly from the geometric outcroppings, the shape of 
the museum building emerges. 

The aboveground entrance to this facility embedded in the hillside draws 
the visitor from a balcony down a staircase enclosed like the walls of a tight 
canyon. At the first turn, a magnificent and larger-than-life-sized bronze 
sculpture of a cougar captivates in its predatory pose. Immediately, the tone is 
set - awe and respect for those more powerful than we, with an undertone of 
foreboding. 

At the bottom of the stairs one finds oneself in an atrium overlooking the 
National Elk Refuge across the highway. In stark contrast to the recent descent, 
the ambiance is one of crisp clean distillation of nature’s energy and power. 
Natural light highlights the textures of the stone facade of the walls. Animal 
tracks emerge from the sandy, distressed sheet-rocked floors. As one looks back 
up the stairs just descended, the dark puma peers down from the perch with 
muscles taught and ready to pounce. The hardiest of visitors responds to the 
gentle push and embarks on the exploration of the galleries. 



National Museum of Wildlife Art ! 
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Part Two: Summit 

Take-Aways 



Through the course of the conversations at the Teton Summit, several 
themes emerged. As participants explored the topics related to environmental 
education program evaluation, particular messages were repeated. These 
themes, or ‘take aways’ are described here for the purpose of summarizing and 
touching on a few of the more prevalent issues that fed the discussions at the 
Summit. Hopefully, these motifs will be revisited and will continue to be fleshed 
out as the exploration of nonformal environmental education program evalua- 
tion continues. 

The four primary take-aways from the Teton Summit 
were: 

1. The importance of carefully matching evaluation approaches to individual 
programs 

2. Understanding the relationship between evaluation and research in environ- 
mental education 

3. The need to infuse evaluation thinking into our environmental education 
programming 

4. Taking time to accurately and carefully identify what the ‘it' is that we are 
evaluating 

Matching 

The first, and most prevalent, theme from the Summit discussions was that 
of ‘matching.’ Each program, even within a single field such as nonformal 
environmental education, has a unique context. That is, the host of variables 
that created the need for a program and that program’s niche in the environ- 
mental education community create the setting for that program and affect 
every aspect of the program design and implementation. These same factors also 
define the evaluation methods and approaches that are most compatible with 
and effective for collecting evaluation information about that program. It is 
often not feasible to design pre and post tests for the adult participants at a 
weekend birding retreat. Adult learners may not appreciate the intrusion of 
formal testing in a nonformal atmosphere. Similarly, a program designed for 
school children may not include an appropriate amount of time for testing 
knowledge before and after the learning experience. Other, less intrusive 
measures may be necessary for the evaluator wanting to assess learning in 
program participants. 



Another equally guiding force in the evaluation of programs, stems from the 
underlying purpose of program evaluation. The reasons why one evaluates are 
many, and the varied paths that programmers take to the evaluation precipice 
bring them to a common point: program evaluation is conducted in order to 
provide the information necessary to make decisions. This desired information 
and the intended uses of the evaluation data drive all of the other elements in 
the evaluative picture. Program evaluation can take many forms, depending on 
the nature of that needed information. 

Good program evaluation does not prescribe a particular research design. It 
does not imply a particular, detailed set of methods. Good program evaluation 
does look at a program with ‘new glasses.’ It does seek out the best way to 
provide the needed information to the stakeholders in the process. However, 
how does one define ‘best?’ The criteria for evaluation indicate that the data 
collected are to be useful. These criteria also state that the data should be 
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accurate. Michael Patton indicated that these criteria, as well as those of 
propriety and feasibility, are both important; however, utility is the first of the 
evaluation criteria and accuracy is commonly listed last. The implied caution for 
program evaluators is not to fall on the sword of sound (methodological) 
research. While not disregarding accuracy, evaluators must place first the need 
to provide useful information and facilitate the use of those data. Good program 
evaluation will not be judged by how large the sample was or where the results 
are published as is the case with most research. Good evaluation will provide 
data that are useful to and used by the stakeholders in the evaluation process. 

In the mix of matching methods, quantitative and qualitative, to programs 
of all types, issues of validity and reliability (accuracy) cannot be ignored of 
course. Methods tailored to a particular environmental education program will 
also require validation. An evaluator cannot determine that the program has 
increased learning in the participants based on how many students expressed 
enthusiasm or smiled during the program. However, the process of seeking a 
previously developed research instrument with established validity and reliability 
may not be the answer either. Validation in evaluation may sometimes be 
likened to the criteria for validity described by Lincoln and Cuba (in Denzin and 
Lincoln, 1994) for qualitative research. Program evaluators can (and should 
with quantitative measures) test their instruments for validity and reliability, but 
the ultimate test comes through ground-truthing the methods. The methods 
used to gather evaluation information must ultimately make sense to those who 
need to use the data collected. Using a stakeholder group to determine if the 
questions being asked in an evaluation will gather the information needed 
ultimately grounds and validates much of the evaluation methodology. This 
process, in addition to using multiple methods, will allow the statements one 
bases on evaluation data to be placed in the context unique to the program being 
evaluated. 

Overall, the evaluator must keep an eye on the intended use of the evalua- 
tion information throughout the evaluation process. By allowing the evaluation 
questions and the program context to guide his or her methodology, the 
evaluation can render useful information. Finally, the evaluator can then 
ensure that the evaluation information is used by actively facilitating its use by 
the stakeholders in the process. 



The second theme from the Teton Summit, reminds us that research and 
evaluation can form a happy partnership. Each serves a different purpose, but 
they are related in many ways and compliment each other in practice. While 
research and evaluation often deal with similar issues, it is important to 
remember that we are not called on to answer pressing research questions with 
every evaluation. At the same time there are important implications to perform- 
ing an evaluation without the benefit of relevant research in nonformal environ- 
mental education. 

Evaluation is inherently comparative. The issue is: to what are we compar- 
ing our program? If you do not include multiple stakeholders, each representing 
different perspectives, an evaluation can lose sight of its larger context: its 
setting within the field of environmental education. Gathering data that are 
useful to the program stakeholders who will be making decisions based on them 
is the guiding purpose of program evaluation, as mentioned above. However, the 
value of stakeholders in an evaluation hinges on the triangulation of perspectives 
and insight that accompanies the diverse representation in a stakeholder group. 
Even with a funder, a program administrator, a front-line educator or inter- 
preter, a community representative, an agency board member, and a prospective 
participant on your stakeholder group, a program, and its evaluation, could still 
remain quite isolated from the larger field of environmental education. It is 
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nearsighted of environmental education agencies to forget that often their 
evaluation questions relate directly to the research questions driving the field of 
nonformal environmental education. Ultimate utility can be best served if the 
evaluation has ties to the research in environmental education. 

One way to address this issue is to consider the larger field of environmental 
education as one of the stakeholders in the program evaluation. This does not 
imply that a researcher in environmental education must be asked to formally 
participate in the evaluation. The evaluation team can keep a close eye on the 
emerging literature in the field. Allowing trends in environmental education to 
inform our evaluation questions keeps us in touch with alternative approaches 
and prevents a reinventing of the wheel with each program and evaluation. 
Keeping evaluation connected to the field of environmental education does not 
imply that the data from a program’s evaluation are generalizable to the field; 
however, it does allow for the possibility that the utility of the data to each 
stakeholder includes a utility broader than the individual program. In this way, 
evaluation can inform the field of environmental education and research in the 
field can inform individual evaluations. A true learning organization will strive 
to build this type of feedback into all of its programming, and hence its program 
evaluation. 

To point out the relevance of this feedback in nonformal environmental 
education, consider the history of environmental education. Throughout the 
early part of the 20 th century, different movements emerged, grew philosophi- 
cally, and informed what was ultimately to become environmental education as 
outlined by several sources in the 1960’s and 70’s and in use today. Interpreta- 
tion and nature guiding, outdoor education, nature study, experiential educa- 
tion, and the conservation education movement all contributed to the field of 
environmental education. The environmental education institutions, their 
missions, and programs today encompass the philosophies of each of these 
movements to a greater or lesser extent. Evaluations informed primarily by 
internally derived criteria can be isolated from the broader goals and objectives 
of the field of environmental education. Neglecting research and established 
trends outside an organization’s mission and historical context can foster 
evaluations - and programs - that lack connections to the broader field of 
environmental education. Therefore, it may be important for program leaders to 
consider the implications of how evaluation criteria are chosen for their 
programs. 

A good marriage has strong ties between its partners. They learn and grow 
from each other, while recognizing and appreciating their different roles. 
Evaluation and research, both as disciplines and as individual projects, should 
inform each other through a familiarity with each other’s priorities and findings 
in the field of nonformal environmental education. 




Another idea that repeatedly arose at the Summit was that of infusing 
evaluation into the environmental education organization and its programming. 
In order to create an environment of ongoing program development, implemen- 
tation, and evaluation, two things are necessary. First, the environmental 
education and interpretive staff must work to develop a culture of reflective 
practice. Second, they must foster this environment by building formal 
evaluation capacity in their organization. 

In nonformal education, reflection on what has ‘worked’ and what hasn’t 
and the active incorporation of this information into programming is often 
already in place on an individual basis. The educator in a nonformal environ- 
ment has historically relied on this informal evaluation in the absence of the 
formal assessment afforded by the classrooms of formal educators. However, 
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developing an enriching atmosphere in which this process is shared among staff 
requires that this process be recognized, valued, and encouraged. Reflective 
practice refers to the incorporation of this evaluation thinking into daily 
programming, as well as remembering to step back from the program and take 
time to see the big picture. This process is similar to what formal program 
evaluation offers to environmental education programs. Formal evaluation 
incorporates time for reflection and discussion among stakeholders as to what 
information is needed to make decisions about programming. Formal evalua- 
tion insures that data collection does not interfere with programming, and then 
allows more time for reflection and use of the evaluation results. 



An organization in which staff are aware of the informal ways they collect 
data each day in their programs and in which they discuss how to incorporate 
their observations into program improvement or development has a culture of 
reflective practice. By formalizing some of these elements of good programming, 
the organization can begin to build its capacity for using formal program 
evaluation. This process involves building on and fostering the informal 
evaluation practices already employed by education staff. As an organization 
develops and grows, it can foster learning about evaluation approaches and 
alternatives, and begin to incorporate the language, concepts, and skills of 
program evaluation. This learning process can bring uniformity across the 
organization and allows for better communication of program successes and 
failures. 



Whether the environmental education organization then decides to conduct 
an evaluation of their program using a member of their own staff or contract 
with an outside evaluator, the organization’s staff will be better equipped to 
participate in and use the results of an evaluation if time has been devoted to 
understanding and valuing the information that is readily available from their 
programming. Although this process can be frustrating and challenging, 
allowing a reflective and learning-oriented culture to grow within the environ- 
mental education organization and become formalized will ultimately permit 
mutually reinforcing evaluations and lead to organizational learning and growth. 

Identify Dug the If 7 in Evaluation 

The fourth and final Summit take-away emerged when the Summit 
participants joined a group of teachers who were attending a workshop on the 
Journeys curriculum. Journeys is a place-based curriculum for K-12 classrooms 
and was developed by the Teton Science School. As part of the program, 
graduate students at the Teton Science School visit classrooms to help teachers 
incorporate the activities into their curriculum. 

During the joint session, the Summit participants and Journeys teachers 
discussed program evaluation in the context of the formal classroom and the 
Journeys curriculum. As the teachers and TSS staff described their program, a 
key evaluation issue emerged: the importance of accurately and specifically 
identifying what it is that we are evaluating. How does an evaluator identify the 
qualities that distinguish this program from all others? Programs such as 
Journeys, that are taking place in disparate classrooms and are implemented by 
many different teachers, manifest themselves in unique ways at each site and 
often change over time as well. When looking at a program to determine its 
‘effectiveness,’ what aspects of the program does the evaluator examine? 

The core elements that define the program must be clearly identified prior 
to a program evaluation. When using a stakeholder group, the evaluator can 
elicit a discrete list of core elements from those closest to the program. In a 
sense, the evaluator facilitates a common definition of what the program should 
look like when successfully implemented. With the Journeys program, this 
described both the pieces of a complete Journeys program and characteristics of 
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a successful Journeys teacher. Care must be taken to identify what exactly is 
central to a program for its success. What are the essential goals and desired 
outcomes for a program? What characteristics must be apparent in a program 
for these goals to be achieved? With th z Journeys program, teachers indicated 
that parental buy-in to the program activities was an important part of the core 
of the Journeys program. This core element is not directly related to a goal or 
description of the curriculum implementation; however, through discussing 
what was working and what wasn’t in the field, the stakeholders to the program 
identified a core piece to the program’s potential success. 

Although lists could be made of all of the important elements of a successful 
program, the tendency for many program stakeholders is to want the evaluation 
to render an effectiveness rating of some sort - a map by which to measure in 
concrete terms whether or not a particular manifestation of the program or 
curriculum is successful. One part of identifying core elements of a program is 
understanding that not all programs will have all elements. The power of 
program evaluation lies in its ability to offer a menu of ways of thinking about or 
looking at effectiveness. For instance, the concept of care -taking arose in the 
Journeys discussion. Student stewardship may be an implicit goal of the 
Journeys curriculum and it was achieved by a few teachers who had been 
working with the program for a few years and had expanded it to include 
community litter pick-ups. A noticeable awareness in the students about 
littering in their community was evident, according to one teacher. Does care- 
taking or stewardship then become a core element? Certainly it is desired 
outcome of the program, but is a program ineffective if there is no evidence of 
student stewardship? Program evaluation can provide evidence of each of the 
measurable objectives that stakeholders identify as important to the program’s 
implementation and success. Then these elements can be combined and 
analyzed to explore program effectiveness defined in a number of ways. A local 
program in its first year of implementation may not be ready to be measured 
against student stewardship as an outcome. The stakeholders and evaluator can 
work together to appropriately match measures and core elements to program 
manifestations. The first, and perhaps most important, step in evaluating any 
program is identifying and describing those core elements. 

As the Teton Summit came to a close, the participants had a lot to think 
about. In the months following that weekend retreat, I have reviewed the 
discussions that took place, spoken with some of the participants, and solicited 
input as to what was most meaningful in the Summit experience. I hope to have 
touched on the primary issues, and 
in some way I hope that each of the 
participants can see a part of their 
experience in this document. The 
Summit Take-Aways could go on for 
much longer, but in narrowing to 
four general themes, I believe the 
spirit of the Summit discussion is 
captured. Hopefully, those who take 
the time to read this document - or 
at least peruse its pages - will find 
some kernel of information relevant 
to their cause. Perhaps these pages 
will be a source of inspiration for 
future conversations about how to 
educate and evaluate in a more 
enlightened and informed manner. 



The core elements that define 
the program must be clearly 
identified prior to the evaluation 
of that program. 



The Teton Summit Group: 
Row 3: John Hayes, Jim Akers, 
Patti Reilly, Court Smith, Sue 
Perin, Brian Windrope, Bob Petty, 
Joe Heimlich, Tom Marcinkowski, 
Janet Ady, Thane Maynard, 
Stephanie Pittard, Philis Ault, 
Michael Patton, Liz Hall, Mark 
Everson, Joe Jones 
Row 2: Annie Jack, Sue McGuire, 
Jack Shea, Emmalou Norland, 
Debbie Brainerd, Terry 
McLaughlin, Gina Zwerling 
Row 1: Cindy Somers, Marcella 
Wells, Kate Wiltz, Judith 
Stockdale, Julia and Ruby 
Washburn 
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Summit 



Utility 

• The Utility Standards are intended to ensure that an evaluation will serve the 
practical information needs of intended users. 

Feasibility 

• The Feasibility Standards are intended to ensure that an evaluation will be 
realistic, prudent, diplomatic, and frugal. 

Propriety 

o The Propriety Standards are intended to ensure that an evaluation will be 
conducted legally, ethically, and with due regard for the welfare for those 
involved in the evaluation, as well as those affected by its results. 

Accuracy 

o The Accuracy Standards are intended to ensure that an evaluation will reveal 
and convey technically adequate information about the features that deter* 
mine worth or merit of the program being evaluated. 

Joint Committee on Standards for Educational Evaluation, Saunders chair, 1994 

Guiding Principles for Evaluators 

American Evaluation Association 

Systematic Inquiry 

» Evaluators conduct systematic, data-based inquiries about what is being 
evaluated. 

Competence 

• Evaluators provide competent performance to stakeholders. 

Integrity/Honesty 

» Evaluators ensure the honesty and integrity of the entire evaluation process. 

Respect for People 

• Evaluators respect the security, dignity, and self-worth of the respondents, 
program participants, clients, and other stakeholders with whom they work. 

Responsibilities for General and Public Welfare 

* Evaluators articulate and take into account the diversity of interests 

and values that may be related to the general and public welfare. 

American Evaluation Association Guiding Principles 

for Evaluators, Shadish et al. 1995. 
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Essential Underpinnings 
of Environmental Education 

Systems 

® A system is made up of parts that can be understood separately. 

° A whole is only understood when the relationships among its parts are 
understood. 

Interdependence 

® Human well-being is inextricably bound with environmental quality. 

• We are part of the natural order, and as such, are challenged to recognize the 
ramifications of our interdependence. 

Place 

• Forging connections with, exploring, and understanding immediate sur- 
roundings forms a base of sensitivity, knowledge, and skills for moving out to 
broader issues and larger systems. 

Integration and infusion 

0 The environment is a context for connecting different disciplines. 

0 Environmental education works best when infused across the curriculum. 

Real world roots 

® Direct experience with the environment, environmental issues, and society 
develops knowledge and skills. 

® Investigation, analysis, and problem solving are essential activities. 

Lifelong learning 



0 Critical and creative thinking, decision making, and communication, and 
collaborative learning are essential skills for meaningful learning in school 
and over a lifetime. 



Adapted from Excellence in EE — Guidelines for Learning (K-12). North 
American Association for Environmental Education. Rock Spring, GA, 1999. 
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Six Principles of Interpretation 

Freeman Tilden, Interpreting our Heritage, 1957 

® Any interpretation that does not somehow relate what is being displayed or 
described to something within the personality or experience of the visitor will 
be sterile. 

0 Information, as such, is not Interpretation. Interpretation is revelation based 
upon information. But they are entirely different things. However, all 
interpretation includes information. 

• Interpretation is an art, which combines many arts, whether the materials 
presented are scientific, historical or architectural. Any art is in some degree 
teachable. 

• The chief aim of Interpretation is not instruction, but provocation. 

• Interpretation should aim to present a whole rather than a part, and must 
address itself to the whole man rather than any phase. 

• Interpretation addressed to children (say, up to the age of twelve) should not 

be a dilution of the presentation to adults, but should follow a fundamentally 

different approach. To be at its best it will require a separate program. 
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Appendix 2: Resource Guide 

The following resource list consists of publications that were mentioned by 
participants during the Summit, as well as those found in electronic literature 
databases. In addition, certain sources were explored more intensely and mined 
for publications of interest. These include: 

© the Journal of Environmental Education 
0 the Canadian Journal of Environmental Education 
° the International Journal of Environmental Education and Information 
® the Southern African Journal of Environmental Education 
® the Journal of Interpretation Research 

® the “EPA Environmental Education Resources” web page (http://www.epa.gov/ 
enviroed/resources.html) 

o the “NAAEE Publication Descriptions” web page (http://www.naaee.org/html/ 
pubdescrips.htm) 

° the “American Camping Association Camp Research & Trends” web page 
(http://www.acacamps.org/research/) 

0 Visitor Studies: Theory, Research, and Practice. Selected papers from the 
Visitor Studies Association Conferences 
Sources were selected mostly by their titles, as opposed to abstracts. In this 
way, some important and relevant pieces might have been left out and others, 
not so significant, might have been included. The review did not include a 
search of major themes in the broader databases. It focused on searches at 
specific periodical journals. In this way, it does not refer to many books and 
audiovisual materials. 



E audio and/or visual aid material 
□ book 

© journal article, report, book chapter or book article 
V web page 



2000 

□ North American Association for Environmental Education, 
National Project for Excellence in Environmental Education 
(2000). Guidelines for Excellence: The Initial Preparation of 
Environmental Educators. Washington, DC: North American 
Association for Environmental Education. ISBN: 

188400878X. 

V EdGateway: http://www.edgateway.net/ 

V EE Link: http://eelink.net/ 

▼ Environmental Education and Training Partnership 
(EETAP): http://www.eetap.org/ 

V National Pollution Prevention Center for Higher Education: 
http://www.umich.edu/%7Enppcpub/ 

▼ NatureServe: http://www.natureserve.org/ 

▼ The EETAP Resource Library: http://www-comdev.ag.ohio- 
state.edu/eetap/index.html 

▼ The GLOBE Program: http://www.globe.gov/ 

▼ The VINE (Volunteer-led Investigations of Neighborhood 
Ecology; North American Association for Environmental 
Education) evaluation program. Contact: http://eelink.net/ 
~npeee/html/2_shortnut.html#VINE and http://naaee.org/ 







1999 

□ Brand, Stewart (1999). The clock of the long now: time 
and responsibility . New York, NY: Basic Books. ISBN: 
046504512X 

© Dettmann-Easler, Detra & Pease, James L.. (1999). 
Evaluating the effectiveness of residential environmental 
education programs in fostering positive attitudes toward 
wildlife. The Journal of Environmental Education, 31(1), 
33-39. Washington, DC: Heldref Publications. ISSN: 
00958964 

□ Diamond, Judy (1999). Practical evaluation guide: tools 
for museums and other informal educational settings 
(American Association for State and Local History Book 
Series). Walnut Creek, California: AltaMira Press. ISBN: 
0761989404 

© Environmental Education and Training Partnership 
(EETAP) (1999). Evaluating the Content of Web Sites: 
Guidelines for Educators. Columbus, OH: EETAP Resource 
Library, [available online at http://www-comdev.ag.ohio- 
state.edu/eetap/publications.htm] 

• Heimlich, Joe & Wang, Katy (1999). Evaluating the 
Structure of Web Sites: Guidelines for Educators. Colum- 
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bus, OH: EETAP Resource Library, [available online at http:/ 
/www-comdev.ag.ohio-state.edu/eetap/publications.htm] 

□ Marsh, Paul E. (1999). What does camp do for kids? A meta- 
analysis of the influence of the organized camping 
experience on the self constructs of youth . Unpublished 
Master of Sciences Thesis, Indiana University [available 
online at http://www.acacamps.org/research/ss3.html. 

€> National Environmental Education and Training Founda- 
tion (NEETF) (1999). 1999 report card: environmental 
readiness for the 21st century. The eighth annual national 
report card on environmental attitudes, knowledge, and 
behavior. Washington, DC: NEETF 

□ North American Association for Environmental Education, 
National Project for Excellence in Environmental Educa- 
tion (1999). Excellence in Environmental Education: 
Guidelines for Learning (K-12). . Washington, DC: North 
American Association for Environmental Education. ISBN: 
1884008755 

© North American Association for Environmental Education, 
National Project for Excellence in Environmental Educa- 
tion (1999). Excellence in Environmental Education: 
Guidelines for Learning (K-12) Executive Summary & Self 
Assessment Tool. Washington, DC: North American 
Association for Environmental Education, [available online 
at http://www.epa.gov/enviroed/resources.html; 

PDF938KB]. ISBN: 1884008771. 

© Reid, Ivan (1999). A scale for measuring the impact of a 
primary school environmental education programme on 
parents’ attitudes. International Journal of Environmental 
Education and Information, 18(1), 1-12. London [England]: 
Taylor & Francis. ISSN: 0144-9281 

® Zelezny, Lynnette C. (1999). Educational interventions that 
improve environmental behaviors: a meta-analysis. The 
Journal of Environmental Education, 31(2), 5-14. Washing- 
ton, DC: Heldref Publications. ISSN: 00958964 

1998 

□ Archie, Michele (Editor) (1998). Environmental Education 
in the United States —Past, Present , and Future: Collected 
Papers from the National Environmental Education 
Summit (November 1-5, 1996: Burlingame, California). 
Troy, Ohio: North American Association for Environmental 
Education. ISBN: 1884008534 

© Black, Paul & William, Dylan (1998). Inside the black box: 
raising standards through classroom assessment. Phi Delta 
Kappa, 80(2), 139-44. ISSN: 0031-7217 

® Bogner, Franz (1998). The influence of short-term outdoor 
ecology education on long-term variables of environmental 
perspective. The Journal of Environmental Education, 

29(4), 17-30. Washington, DC: Heldref Publications. ISSN: 
00958964 

@ Bowers, C.A. (1998). A cultural approach to environmental 
education: putting Michael Sanera’s ideology into perspec- 
tive. Canadian Journal of Environmental Education, 3, 57- 

Si audio and/or visual aid material 
□ book 

® journal article, report, book 
chapter or book article 
▼ web page 



66. Whitehorse, Yukon: Canadian Journal of Environmental 
Education. ISSN: 1205-5352 

© Courenay-Hall, Pamela (1998). Textbooks, teachers and full- 
colour vision: some thoughts on evaluating environmental 
education “performance”. Canadian Journal of Environ- 
mental Education , 3, 27-40. Whitehorse, Yukon: Canadian 
Journal of Environmental Education. ISSN: 1205-5352 

© Disinger, John (1998). Environmental education’s defini- 
tional problem. In Harold R. Hungerford (Editor). Essential 
Readings in Environmental Education (pp. 17-32). 
Champaign, Illinois: Stipes Publishing Co. (original article: 
1983, ERIC . Information Bulletin #2). ISBN: 0875637566 

© Environmental Protection Agency (1998). Catalog ofEPA 
Environmental Education Materials and Resources. 
Washington, D.C.: U.S. Environmental Protection Agency, 
Office of Communications, Education and Media Relations. 
NSCEP #171-B-98-003 

© Gerakis, Argyrios (1998). Evaluating adult groundwater 
education. The Journal of Environmental Education, 30(1), 
20-24. Washington, DC: Heldref Publications. ISSN: 
00958964 

© Heffernan, Bernadette M. (1998). Evaluation techniques for 
the Sandy Point Discovery Center, Great Bay national 
Estuarine Research Reserve. The Journal of Environmental 
Education, 30(1), 25-33. Washington, DC: Heldref Publica- 
tions. ISSN: 00958964 

□ Lieberman, Gerald A. & Hoody, Linda L. (1998). Closing the 
achievement gap: using the environment as an integrating 
context for learning . San Diego, California: State Education 
and Environment Roundtable. LCCN: 99163317 

1 Lieberman, Gerald A. (1998). Closing the achievement gap: 
using environment as an integrating context in K-12 
education: a video summary. San Diego, California: West 
Peak Media (videocassette; 14 min.). Access to text: http:// 
members.home.net/spawnl/sciwiz/ExecSumm.pdf 

© National Environmental Education and Training Founda- 
tion (NEETF) (1998). The national report card on environ- 
mental knowledge, attitudes and behaviors: the Seventh 
Annual Survey of Adult Americans. Washington, DC: 

NEETF 

© Norris, Kimberly S. & Jacobson, Susan K. (1998). Content 
analysis of tropical conservation programs: elements of 
success. The Journal of Environmental Education, 30(1), 
38-44. Washington, DC: Heldref Publications. ISSN: 
00958964 

® Rome, Abigail & Bart, Romero (1998). Enhancing conserva- 
tion education opportunities in nature reserves in Tropical 
countries: a case study in Belize. The Journal of Environ- 
mental Education, 30(1), 34-37. Washington, DC: Heldref 
Publications. ISSN: 00958964 

• Sanera, Michael (1998). Environmental education: promise 
and performance. Canadian Journal of Environmental 
Education, 3, 9-26. Whitehorse, Yukon: Canadian Journal of 
Environmental Education. ISSN: 1205-5352 

• Stewart, Emma J.; Hayward, Bronwyn M.; & Devlin, Patrick 
J. (1998). The “place” of interpretation: a new approach to 
the evaluation of interpretation. Tourism Management, 
19(3), 257-266. New York, NY: Elsevier Science. ISSN: 0261- 
5177 
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© Tomsen, Jennifer L. & Disinger, John F. (1998). A method 
for assessing effects of an introductory environmental 
history course on student worldviews. The Journal of 
Environmental Education , 29(2), 11-20. Washington, DC: 
Heldref Publications. ISSN: 00958964 

□ Volk, Trudi & McBeth, William (1998). Environmental 
Literacy in the United States: What Should Be...What 
Is.. .Getting from Here to There. Washington, DC: North 
American Association for Environmental Education. ISBN: 
1884008739. 

□ Volk, Trudi L. & McBeth, William C. (1998). Environmental 
literacy in the United States: what should be. . . What is. . . 
Getting from here to there. Rock Spring, GA : North 
American Association for Environmental Education. ISBN: 
1884008739 

© Winter, Patricia L.; Cialdini, Robert B.; Bator, Renee J.; 
Rhoads, Kelton; & Sagarin, Brad J. (1998). An analysis of 
normative messages in signs at recreation settings. The 
Journal of Interpretation Research , 3(1). Fort Collins, CO: 
National Association for Interpretation. ISSN: 10925872 

1997 

© Bachiorri, Antonella & Vezzosi, Monica (1997). Outcome 
research in environmental education: a case study. Interna- 
tional Journal of Environmental Education and Informa- 
tion, 16(2), 141-150. London [England]: Taylor & Francis. 
ISSN: 0144-9281 

© Campbell, Heather & Wells, Marcella (1997). Assessment of 
museum world wide web home page formats. In Marcella 
Wells & Ross J. Loomis (Editors), Visitor Studies: Theory, 
Research , and Practice. Selected papers from the 1996 
Visitor Studies Association Conference, 9(1), 216-226. 
Jacksonville, Alabama: Visitor Studies Association. ISSN: 
10645578 

@ Casaleiro, P. (1997). Evaluating the moving dinosaurs: 
surveys of the blockbuster exhibition in four European 
capital cities. In Marcella Wells & Ross J. Loomis (Editors), 
Visitor Studies: Theory, Research, and Practice. Selected 
papers from the 1996 Visitor Studies Association Confer- 
ence, 9(1), 157-170. Jacksonville, Alabama: Visitor Studies 
Association. ISSN: 10645578 

® Dierking, L. D. (1997). Pacific Science Center’s Science 
Carnival Consortium Project: a qualitative evaluation. In 
Marcella Wells & Ross J. Loomis (Editors), Visitor Studies: 
Theory, Research, and Practice. Selected papers from the 
1996 Visitor Studies Association Conference, 9(1), 28-39. J 
Jacksonville, Alabama: Visitor Studies Association. ISSN: 
10645578 

@ Fischer, Daryl (1997). Visitor panels: in-house evaluation of 
exhibit interpretation. In Marcella Wells & Ross J. Loomis 
(Editors), Visitor Studies: Theory, Research, and Practice. 
Selected papers from the 1996 Visitor Studies Association 
Conference, 9(1), 51-62. Jacksonville, Alabama: Visitor 
Studies Association. ISSN: 10645578 

• Gough, Annette (1997). Evaluation of Australian govern- 
ment literature on the environment. The Journal of 
Environmental Education, 28(4), 18-25. Washington, DC: 
Heldref Publications. ISSN: 00958964 

• Haas, N. T. (1997). Project Explore: how children are really 
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learning in children’s museums. In Marcella Wells & Ross J. 
Loomis (Editors), Visitor Studies: Theory, Research, and 
Practice. Selected papers from the 1996 Visitor Studies 
Association Conference, 9(1), 63-69. Jacksonville, Alabama: 
Visitor Studies Association. ISSN: 10645578 

□ Hollweg, Karen (Ed.) (1997). Are We Making a Difference ?: 
Lessons Learned from Vine Program Evaluations. Washing- 
ton, DC: North American Association for Environmental 
Education. ISBN: 188400847X 

© Jacobson, Susan K. (1997). Rapid Assessment for Conserva- 
tion Education (RACE). The Journal of Environmental 
Education, 28(3), 10-19. Washington, DC: Heldref Publica- 
tions. ISSN: 00958964 

© Jennings, H. (1997). Focus groups with zoo visitors who are 
blind or have low vision: how can we deliver our messages 
to those who cannot see signs? In Marcella Wells & Ross J. 
Loomis (Editors), Visitor Studies: Theory, Research, and 
Practice. Selected papers from the 1996 Visitor Studies 
Association Conference, 9(1), 171-175. Jacksonville, 
Alabama: Visitor Studies Association. ISSN: 10645578 

O Jensen, Patricia & Hayward, Jeff (1997). Strategic planning 
in the woods: applying the results of audience research. In 
Marcella Wells & Ross J. Loomis (Editors), Visitor Studies: 
Theory, Research, and Practice. Selected papers from the 
1996 Visitor Studies Association Conference, 9(1), 70-78. 
Jacksonville, Alabama: Visitor Studies Association. ISSN: 
10645578 

© Kirchberg, Volker (1997). Mapping museums’ catchment 
areas: a tool for understanding museum visitorship. In 
Marcella Wells & Ross J. Loomis (Editors), Visitor Studies: 
Theory, Research, and Practice. Selected papers from the 
1996 Visitor Studies Association Conference, 9(1), 79-94. 
Jacksonville, Alabama: Visitor Studies Association. ISSN: 
10645578 

□ Knaap, Gerrit J. & Kim, Tschangho J. (1997). Environmen- 
tal Program Evaluation: A primer. Urbana and Chicago, 
Illinois: University of Illinois Press. ISBN: 025202334X 

□ McMillan, James H. & Schumacher, Sally (1997). Research 
in education: a conceptual introduction (4 th . Ed.). New 
York, NY: Longman. ISBN: 0673997413 

© Morgan, J. Mark, Absher, James, Loudon, Bob & Sutherland, 
Dave (1997). The relative effectiveness of interpretive 
programs directed by youth and adult naturalists in a 
national forest. The Journal of Interpretation Research, 

2(1). Fort Collins, CO: National Association for Interpreta- 
tion. ISSN: 10925872 

□ National Environmental Education and Training Founda- 
tion (NEETF) (1997). The national report card on environ- 
mental knowledge, attitudes and behaviors. Washington, 
DC: NEETF 

© Nelson, K. (1997). Evaluation of a naturalistic exhibit: the 
northern trail at Woodland Park Zoological Gardens. In 
Marcella Wells & Ross J. Loomis (Editors), Visitor Studies: 
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Theory, Research, and Practice. Selected papers from the 
1996 Visitor Studies Association Conference, 9(1), 95-102. 
Jacksonville, Alabama: Visitor Studies Association. ISSN: 
10645578 

© Poole, J. L.; Cota, A.; & Loomis, R. (1997). Front-end 
evaluation and planning changes for a community 
museum. In Marcella Wells & Ross J. Loomis (Editors), 
Visitor Studies: Theory, Research, and Practice. Selected 
papers from the 1996 Visitor Studies Association Confer- 
ence, 9(1), 185-194. Jacksonville, Alabama: Visitor Studies 
Association. ISSN: 10645578 

© Simmons, Dan (1997). TheNAAEE Standards Project: 
papers on the development of environmental education 
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American Association of Zoological Parks 
and Aquariums 
AZA 

Contact Information: 

Address: 

8403 Colesville Rd, Suite 710 
Silver Spring, MD 20910-3314 
Phone: (301) 562-0777 
Fax: (301) 562-0888 

Web site: 

http://www.aza.org/ 

EE-related section: 

Department of Conservation Education 
http://www.aza.org/dept/educ/ 

Publications’ web page: 

http://www.aza.org/Communique/01-01/index.htm 
Periodic Publication: 

Communique (online/paper magazine) 

Annual meeting: 

AZA Workshops 
AZA Annual Conference 



American Evaluation Association 
AEA 

Contact Information: 

Address: 

505 Hazel Circle 
Magnolia, AR 71753 
Phone: 888-232-2275 or 870-234-7433 
Email: AEA@kistcon.com 

Web site: 

http://www.eval.org 

EE-related section: 

Topical Interest Group: Environmental Program 
Evaluation 

Publications’ web page: 

http://www.eval.org/AEADocuments/documents.htm 
http://www.eval.org/Publications/publications.html 
Periodic Publication: 

New Directions for Evaluation 
American Journal of Evaluation 
Annual meeting: 

Annual Conference 



American Camping Association 
ACA 

Contact Information: 

Address: 

5000 State Road 67 North 
Martinsville, IN 46151-7902 
Phone: 765-342-8456 
Fax: 765-342-2065 

Web site: 

http://www.acacamps.org/ 

Publications’ web page: 

http ://www.acacamps.org/research/ 

http://www.acacamps.org/research/bib/ 

http://www.acacamps.org/knowledge/ 

http://www.acacamps.org/bookstore/ 

http://www.acacamps.org/campmag/ 

http://www.acacamps.org/campline/ 

Periodic Publication: 

Camping Magazine (magazine, online/paper) 
CampLine (newsletter, online/paper) 

Annual meeting: 

ACA National Conference 
ACA Regional Conferences 
Local Conferences 
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American Folklore Society 
AFS 

Contact Information: 

Address: 

4350 N. Fairfax Drive, Suite 640 
Arlington, VA 22203 

Web site: 

http://www.afsnet.org/ 

Publications’ web page: 

http://www.afsnet.org/publications/ 

Periodic Publication: 

AFSNews (newsletter, paper/online) 

Journal of American Folklore (table of contents 
online, quarterly journal) 

Annual meeting: 

AFS Annual Meeting 

Association of Nature Center 
Administrators 
ANCA 

Contact Information: 

Address: 

Charity Krueger 

Aullwood Audubon Center 

1000 Aullwood Road 

Dayton, OH 45414 

Phone: 1-800-490-ANCA 

Fax: 937-890-2382 

E-mail: ckrueger@audubon.org 
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Web site: 

http://www.natctr.org/index.htm 

Publications’ web page: 

http://www.natctr.org/natctr.hand.htm 
Periodic Publication: 

Directions Newsletter 

Annual meeting: 

Annual Summit for Nature Center and Residential 
Environmental Learning Center Administrators 

Association of Science-Technology Centers 
Incorporated 
ASIC 

Contact Information: 

Address: 

1025 Vermont Avenue NW Suite 500 
Washington, DC 20005-3516 
Phone: 202/783-7200 
Fax: 202/783-7207 
E-mail info@astc.org 

Web site: 

http://www.astc.org 
Publications’ web page: 

http://www.astc.org/pubs/index.htm 
http://www.astc.org/resource/educator/educmain.htm 
http://www.astc.Org/pubs/cataIog.htm#education 
Periodic Publication: 

ASTC Dimensions 
Annual meeting: 

ASTC Annual Conference 



Wilderness Education Association 
WEA 

Contact Information: 

Address: 

PO Box 158897 
Nashville, TN 37215 
Phone: (615) 53151 74 
Fax: (615) 331 9020 
Email: wea@edge.net 

Web site: 

http://wiIdernesseducation.org/ 



Worth American Association for 
Environmental Education 
WAAEE 

Contact Information: 

Address: 

1825 Connecticut Ave NW, 8th Floor 
Washington DC 20009-5708 
Phone (202) 884-8912 
Fax (202) 884-8455 
E-mail: CSmith409@aol.com 

Web site: 

http://naaee.org/ 

Publications’ web page: 

http://naaee.org/html/pubintro.htm 
Periodic Publication: 

Environmental Communicator 
Annual meeting: 

Annual Conference 



National Association for Interpretation 
NAI 

Contact Information: 

Address: 

P.O. Box 2246 
Fort Collins, CO 80522 or 
528 South Howes 
Fort Collins, CO 80521 
Phone: (970) 484-8283 
Fax: (970) 484-8179 
Toll-free: (888) 900-8283 

Web site: 

http://www.interpnet.com 

Publications’ web page: 

http://www.interpnet.com/interpnet/store/next.htm 
Periodic Publication: 

Journal of Interpretation Research 
Legacy 

InterpNews (newsletter) 

Annual meeting: 

National Interpreters Workshop 
Regional Workshops 



Visitors Studies Association 
VSA 

Contact Information: 

Address 

Department of Psychology 
Colorado State University 
Fort Collins, CO 805233-1876 
Phone/Fax: (970) 491-4352 

Web site: 

http://www.museum.msu.edu/vsa/ 

Publications’ web page: 

http://www.museum.msu.edu/vsa/pagelO.html 

Periodic Publication: 

Visitor Studies Today! 

Annual meeting: 

Annual Visitor Studies Conference 
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